{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-07T02:19:02.272939Z",
     "start_time": "2019-04-07T02:19:02.268259Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "***\n",
    "***\n",
    "\n",
    "# Introduction to Neural Network\n",
    "\n",
    "\n",
    "***\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-07T05:41:09.971531Z",
     "start_time": "2019-04-07T05:41:09.967172Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "\n",
    "<img src= 'img/neuralnetwork/playground.png' width= \"700px\">\n",
    "\n",
    "http://playground.tensorflow.org/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## House Price\n",
    "\n",
    "Let’s start with a simple example. \n",
    "- Say you’re helping a friend who wants to buy a house.\n",
    "\n",
    "- She was quoted $400,000 for a 2000 sq ft house (185 meters). \n",
    "\n",
    "Is this a good price or not?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "So you ask your friends who have bought houses in that same neighborhoods, and you end up with three data points:\n",
    "\n",
    "\n",
    "\n",
    "| Area (sq ft) (x)  |  Price (y)   |  \n",
    "| -------------|:-------------:|\n",
    "|2,104|399,900|\n",
    "|1,600|329,900|\n",
    "|2,400|369,000|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "\n",
    "$$y = f(X) = W X$$\n",
    "\n",
    "- Calculating the prediction is simple multiplication.\n",
    "- But before that, we need to think about the weight we’ll be multiplying by. \n",
    "- “training” a neural network just means finding the weights we use to calculate the prediction.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "A simple predictive model (“regression model”)\n",
    "- takes an input, \n",
    "- does a calculation, \n",
    "- and gives an output \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src= 'img/neuralnetwork/data_points_graph_animated.gif' width= \"700px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-07T03:15:14.317623Z",
     "start_time": "2019-04-07T03:15:14.313438Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Model Evaluation\n",
    "- If we apply our model to the three data points we have, how good of a job would it do?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "\n",
    "<img src= 'img/neuralnetwork/data_points_error_animated.gif' width= \"700px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "\n",
    "<img src= 'img/neuralnetwork/model_evaluation.png' width= \"500px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**Loss Function (also, cost function)**\n",
    "\n",
    "- For each point, the error is measured by the difference between the **actual value** and the **predicted value**, raised to the power of 2. \n",
    "- This is called **Mean Square Error**. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "\n",
    "<img src= 'img/neuralnetwork/lines_and_errors_animated.gif' width= \"700px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "- We can't improve much on the model by varying the weight any more. \n",
    "- But if we add a bias (intercept) we can find values that improve the model.\n",
    "\n",
    "<img src= 'img/neuralnetwork/NNs_bias_2.png' width= \"500px\">\n",
    "\n",
    "$$y = 0.1 X + 150$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**Gradient Descent**\n",
    "\n",
    "- Automatically get the correct weight and bias values \n",
    "- minimize the loss function.\n",
    "\n",
    "<img src= 'img/neuralnetwork/gd.png' width= \"700px\">\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Regression\n",
    "\n",
    "<img src= 'img/neuralnetwork/NNs_2_variables.png' width= \"500px\">\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Classification\n",
    "\n",
    "<img src= 'img/neuralnetwork/softmax-regression-scalargraph.png' width= \"500px\">\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## softmax\n",
    "\n",
    "The softmax function, also known as softargmax or normalized exponential function, is a function that takes as input a vector of K real numbers, and normalizes it into a probability distribution consisting of K probabilities. \n",
    "\n",
    "$$softmax = \\frac{e^x}{\\sum e^x}$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-06-21T03:08:08.347877Z",
     "start_time": "2019-06-21T03:08:08.340327Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import torch\n",
    "from torch import nn, optim\n",
    "from torch.autograd import Variable\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-06-21T03:08:54.348801Z",
     "start_time": "2019-06-21T03:08:54.341059Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.02364054, 0.06426166, 0.1746813 , 0.474833  , 0.02364054,\n",
       "       0.06426166, 0.1746813 ])"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def softmax(s):\n",
    "    return np.exp(s) / np.sum(np.exp(s), axis=0)\n",
    "\n",
    "softmax([1.0, 2.0, 3.0, 4.0, 1.0, 2.0, 3.0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-06-21T03:09:07.460322Z",
     "start_time": "2019-06-21T03:09:07.309428Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAETCAYAAAAh/OHhAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3Xt8VWed7/HPLzsJECBcQ6HcC6GAvWFTeoN6a5XWjjgvq0U9Xmac06ljq85xxtsZHS8zL0ed8bRqlenRzhxHbdVaKzoojjrVpGiFQi8ChaQUSgqU7HC/5rJ/54+1A4tNaLJCkmdfvu/XK6+sy7PX/rFL95dnrWc9y9wdERGR3ioLXYCIiBQWBYeIiCSi4BARkUQUHCIikoiCQ0REElFwiIhIIgoOERFJRMEhIiKJKDhERCSR8tAFDITx48f7jBkzQpchIlJQHn/88bS71/TUriiDY8aMGaxduzZ0GSIiBcXMtvemnU5ViYhIIgoOERFJRMEhIiKJKDhERCQRBYeIiCSi4BARkUQUHCIikoiCQ0SkSNz9y0ZWN6UH/H0UHCIiRWDvkTbu+tUW1m7fN+DvpeAQESkCjzalcYdFteMH/L0UHCIiRaChMc3IoeVcMnnUgL+XgkNEpMC5Ow1Naa6ZNY7y1MB/rSs4REQK3HPpI7yw/xiLa3uc2LZfKDhERApcfWM0kmrxIFzfAAWHiEjBq29MM3XsMKaPGz4o7xc8OMxsiZltNrMmM/voWdq80syeMLMNZvabwa5RRCRftXdm+P3W1kE7TQWBH+RkZingHuAGoBlYY2Yr3H1jrM1o4GvAEnd/3swmhKlWRCT/PLFjP4dPdLB49uCcpoLwPY6FQJO7b3X3NuABYGlOm7cBD7n78wDuvmeQaxQRyVv1jWnKDK6ZVTrBMRnYEVtvzm6LmwOMMbNHzOxxM3tndwcys9vMbK2ZrW1paRmgckVE8ktDYwsXTxnNqKqKQXvP0MFh3WzznPVy4HLg9cDrgE+Y2ZwzXuR+r7vXuXtdTc3gnesTEQnlwLF2nmw+wHWDNJqqS9BrHEQ9jKmx9SnAzm7apN39CHDEzH4LXApsGZwSRUTy0++ebaUz4ywaxOsbEL7HsQaoNbOZZlYJLANW5LT5MbDYzMrNrAq4Etg0yHWKiOSdhqYWqipTLJg2ZlDfN2iPw907zOwOYBWQAu5z9w1mdnt2/3J332RmPweeAjLAN9z9j+GqFhHJDw2Naa66YByV5YPbBwh9qgp3XwmszNm2PGf9i8AXB7MuEZF8tmPvUba1HuVd18wY9PcOfapKRET6YLCnGYlTcIiIFKCGphYmVg9lVs2IQX9vBYeISIHpzDiPNrWyuHY8Zt3d1TCwFBwiIgXm6RcOcOBY+6A87a87Cg4RkQLT0BjNjnHtIN+/0UXBISJSYOob08yfVM34EUOCvL+CQ0SkgBw50cG65/exeE6Y3gYoOERECspjz7XS3uksnh1uTj4Fh4hIAalvTDOkvIy6GYM7zUicgkNEpIA0NKZZOHMsQytSwWpQcIiIFIhdB47RuOdwkLvF4xQcIiIFoiE7zciigNc3QMEhIlIwGprSjB9RydyJI4PWoeAQESkAmYzzaFOaRbPHU1Y2+NOMxCk4REQKwKbdB0kfbmNRbfhHYys4REQKwKnrG2EvjIOCQ0SkIDQ0pZlz3ggmjhoauhQFh4hIvjve3skfntsbfDRVFwWHiEieW7NtLyc6MsHv3+ii4BARyXMNjWkqUsaVF4wNXQqg4BARyXv1jWkunz6Gqsry0KUACg4RkbzWcugEG3cdZHEeDMPtEjw4zGyJmW02syYz+2g3+19pZgfM7InszydD1CkiEsLqZ/NnGG6XoP0eM0sB9wA3AM3AGjNb4e4bc5rWu/vNg16giEhg9Y1pRg2r4KLJo0KXclLoHsdCoMndt7p7G/AAsDRwTSIiecHdaWiMphlJBZ5mJC50cEwGdsTWm7Pbcl1tZk+a2c/M7GWDU5qISFhNew6z++BxFuXJMNwuoS/RdxehnrO+Dpju7ofN7CbgYaD2jAOZ3QbcBjBt2rT+rlNEZNDV59E0I3GhexzNwNTY+hRgZ7yBux9098PZ5ZVAhZmd8Sm6+73uXufudTU1+TP6QESkrxqa0swcP5ypY6tCl3Ka0MGxBqg1s5lmVgksA1bEG5jZRDOz7PJCoppbB71SEZFB1NaR4fdbW/OutwGBT1W5e4eZ3QGsAlLAfe6+wcxuz+5fDtwCvNfMOoBjwDJ3zz2dJSJSVNY9v4+jbZ15d30Dwl/j6Dr9tDJn2/LY8leBrw52XSIiITU0pkmVGVfPGhe6lDOEPlUlIiLdqG9Kc9nU0VQPrQhdyhkUHCIieWb/0Taeat6fl9c3QMEhIpJ3Vj/bijt5M416LgWHiEieqW9MM3JIOZdOHR26lG4pOERE8oi7U9/YwlWzxlGRys+v6PysSkSkRG1vPUrzvmN5e5oKFBwiInmlvik/pxmJU3CIiOSRhsYWJo8exszxw0OXclYKDhGRPNHRmWF1UyuLa8eTnWkpLyk4RETyxJPNBzh0oiMvpxmJU3CIiOSJhsY0ZnDtLAWHiIj0QkNTCxdPHsWY4ZWhS3lJCg4RkTxw6Hg7657P32lG4hQcIiJ54Pdb99KZ8by/vgEKDhGRvNDQ2MKwihSXTx8TupQeJQoOM1vcy3Yf6Fs5IiKlqb4pzZUXjGVIeSp0KT1K2uP4tZn93dl2mtkYM/sx8KVzK0tEpHS8sP8YW1uOFMT1DUgeHE3Ap83sl2Y2Mb7DzK4BngD+BHi4n+oTESl6DY0tAFw3pyZwJb2TNDguB74NvBpYb2Y3AJjZx4BHgAnAHe7+pv4sUkSkmNU3pjmvegi1E0aELqVXEj1z3N2PAu8ys18B9wA/M7ONwMuALcCt7v5U/5cpIlKcMhnn0aY0r5o7Ia+nGYlLFBxd3P1bZjYS+ApwEdACXOfuLf1ZnIhIsduw8yD7jrbn9TTquRIPxzWzMjP7R+Bu4DCwGqgBHjGzi/u5PhGRolbfFP17+9oCuTAOyYfjTgV+C3wMeBqoc/dFwP8GaoHHzOyvEh5ziZltNrMmM/voS7S7wsw6zeyWJMcXEclnDY1p5k4cyYSRQ0OX0mtJexxPANcAXweucvctAO7+OeCVRKesvmJmD/XmYGaWIrpWciMwH3irmc0/S7vPA6sS1isikreOtXWydtu+gjpNBcmDowy4xd3f5+4n4jvcfTVwKbACWNrL4y0Emtx9q7u3AQ+c5bV3Aj8E9iSsV0Qkbz32XCttnRkW1RbGMNwuSS+OL3D3bWfb6e77gT81szt6ebzJwI7YejNwZbyBmU0G/pRoCPAViaoVEcljDY1pKsvLWDhjbOhSEknU43ip0Mhp99VeHrK7sWees34X8BF373zJA5ndZmZrzWxtS4sGd4lI/qtvTHPFjDEMq8z/aUbiQk9y2AxMja1PAXbmtKkDHjCzbcAtwNfM7I25B3L3e929zt3ramoKq9snIqVnz8HjbH7xEItmF973VeL7OCy6Q+UW4HVEp5qGdNPM3f01vTjcGqDWzGYCLwDLgLflHGhm7L3/Hfipu2tKExEpaA1NaYCCuzAOCYPDzIYAK4lGUBnRaaX46SaPbe+Ru3dkr4esAlLAfe6+wcxuz+5fnqQ+EZFC0dCYZtzwSuZPqg5dSmJJexwfAV4FfBb4MtHw208B9xKFyT8R3RD4jt4e0N1XEoVRfFu3geHu705Yr4hI3nF36pvSXDN7PGVlhTHNSFzSaxxvBta5+9+7e2vXRnff7e4PEI18uhn4YD/WKCJSVDa/eIiWQydYXEB3i8clDY5ZwKOxdQcqTq64bwX+E3j3OVcmIlKkGhqj6xuF8JjY7iQNjnbgeGz9ENE8VXHbgQvOpSgRkWJW35hmVs1wzh89LHQpfZI0OJqJRlJ12QJcndNmAbD3XIoSESlWx9s7eey5VhYX2N3icUmD41Giuaq6PAxcbGbfNLPXm9kXgeuJHuokIiI51m3fx/H2TEEOw+2SdFTVd4GpZjYjexf5XURzS/0Z0XUNI3q87FlnuRURKWX1TWnKy4wrLxgXupQ+S/oEwEeI9Sbc/aiZXUsUHrOBbcBPsk8KFBGRHPWNLbx82hhGDOnTc/TywjlX7u4dRDPXiojIS9h7pI0NOw/y19fPCV3KOQk9V5WISMl4tCmNe2FOMxLXpx6HmV1C9OyNKcTu44hxd//suRQmIlJsGhrTVA8t55Ipo0OXck6SzlU1FvgPYEnXprM0daJpSUREhOw0I40tXDNrPKkCnGYkLmmP4y6ix7z+Evg20Yy2Hf1dlIhIsdmaPsLOA8d536sL+zQVJA+Om4HV7v7agShGRKRYdU0zsrgAn7+RK+nF8RTR7LciIpJAfWOaaWOrmDauKnQp5yxpcKxD81CJiCTS3pnh91tbC3ZSw1xJg+OzwM1mtmggihERKUZP7NjP4RMdXFckwZH0zvFfm9ky4Edm9lOiHsiBs7T9Vj/UJyJS8Oob05QZXD2rBIPDzCqJphcZA7wr+5P7mNiuR8cqOEREiKYZuWTKaEYN6+62t8KTdFTV54jCYiPwPWAnGo4rInJWB4618+SO/bzvVbNDl9JvkgbHMuBp4Ap3bxuAekREisrvnm0l4xT08zdyJb04Phr4hUJDRKR3GppaGF6ZYsG0wp5mJC5pcGwCJg1EISIixai+Mc1VF4yjIlU8c8om/ZP8C/BGM+u3OYHNbImZbTazJjM74wFQZrbUzJ4ysyfMbK2GAotIodix9yjbW48W/Gy4uZJe43gB+DnwmJndDTzO2Yfj/rang5lZCrgHuIHoeeZrzGyFu2+MNfsVsMLdPTsr7/eBuQnrFhEZdPXZaUYWFdH1DUgeHI8QDbU14JOcORQ3LtWL4y0Emtx9K4CZPUA03PdkcLj74Vj74T28p4hI3qhvbGHSqKHMqhkeupR+lTQ4PkP/fnFPBnbE1puBK3MbmdmfEg0FngC8vh/fX0RkQHRmnNXPtvLa+edhVtjTqOdKeuf4p/r5/bv7NM8IJnf/EdHd6tcRTXty/RkHMrsNuA1g2rRp/VymiEgyT79wgAPH2lk8p7hOU0HCi+NmNs3MqntoM9LMevvN3QxMja1PIbqpsFvZ6yazzOyMK03ufq+717l7XU1N8f2HEpHC0tDYAsC1s8YFrqT/JR1V9RzwgR7avD/brjfWALVmNjM7nckyYEW8gZnNtmw/z8xeDlQCrYmqFhEZZL9tTPOy86sZN2JI6FL6XdJrHMbZHxebmLt3mNkdwCqii+n3ufsGM7s9u3858CbgnWbWDhwDbnV3XSAXkbx15EQH65/fx3sWFedTKJIGR2+cBxzpbWN3XwmszNm2PLb8eeDz/VadiMgAe+y5Vto7veju3+jSY3CY2TtzNl3WzTaIegzTgHcQzWclIlKS6hvTDCkv4/LpY0KXMiB60+P4d06NdHKi+yyWdtOu6xTWUeDT51yZiEiBqm9Ms3DmWIZW9OZ2tsLTm+D4c07d9Hcf8DDw427adRJdtP6du+/vtwpFRArIrgPHaNpzmFvrpvbcuEC9ZHBkh95+t2s2XDN7F/Cwnu4nItK9hpPTjBTn9Q3oeTjuPuAjsfVtgHoTIiJnUd+YZvyIIcydODJ0KQOmp+DwnDbvAi4buHJERApXJuM82pRmce34optmJK6n4NgFFM/zDkVEBtCm3QdpPdLGotnFe5oKer44/mvg7dkpPnZlt73RzGb08Dp39/ecY20iIgWlFK5vQM/B8WGiG/puIOqdONGpqp5OVzmg4BCRklLfmGbOeSM4r3po6FIG1EsGh7u/CCwxswqiR8ZuA+4C7h740kRECsfx9k7+sG0v77hqeuhSBlyvphxx93bgeTPbDmxz9+0DW5aISGFZs20vbR2Zoj9NBcmfxzFzoAoRESlk9Y1pKlNlXDlzbOhSBlyfJzk0s0XAAmA00XPH17l7Q38VJiJSSOob01w+fQxVlQMxd2x+SfwnzD4T49vAhV2byM5lZWabgXe6+9p+q1BEJM+1HDrBpl0H+dvXXdhz4yKQKDjMbDbREN1qoCG7vIvowvmrgMXAf5nZQndv7OdaRUTy0upno2G4xTqNeq6kPY5PACOIHqb0g5x9nzKzW4AHgL8justcRKTo/WZLC6OrKnjZ+aNClzIokj469nqiSQ5zQwMAd3+QaObc68+1MBGRQvDkjv385MmdvG7+RFJlxTvNSFzS4BgPPNNDm2ey7UREitrB4+3cef96Jowcysdvmhe6nEGT9FRVCzC/hzZzgXTfyhERKQzuzscfepoX9h/j+395FaOqKkKXNGiS9jh+DbzBzJZ1t9PM3kT0dMBfnmthIiL57HtrdvDTp3bxv26Yw+XTi//ejbikPY7PEAXDd8zsfcB/E42qmgi8ElgEHAL+oR9rFBHJK1tePMSnfrKBRbPH895XzApdzqBLeud4k5ldD3wLuDb70/VYWYDNwLs0FFdEitXx9k7u+O46Rgwp50u3XkpZiVwQj0t8A6C7rwHmmdk1wMuBUUR3jq9390f7uT4Rkbzy6Z9sZMuLh/nWny9kwsjingX3bJJe4zjJ3Ve7+1fd/R+J7t2YYGavNbNUkuOY2RIz22xmTWb20W72v93Mnsr+rDazS/tas4jIufjpUzu5/w/Pc/srZnHdnJrQ5QSTKDjM7L1m9piZjY1tuxzYBDwI/AxYbWbDe3m8FHAPcCPRaK23mlnuqK3ngFe4+yXAZ4F7k9QsItIfduw9ysd++DQLpo3mQ6+dE7qcoJL2OG4lerrf3ti2LwJjgH8DVgJXALf38ngLgSZ33+rubUQ9l6XxBtmezb7s6u+BKQlrFhE5J20dGe64fz0YfHnZAipSfT5ZUxSS/ulrgae6VrKPlH0F8E13/wt3/xNgDfC2Xh5vMrAjtt6c3XY27yHq1ZzBzG4zs7VmtralpaWXby8i0rN/+cVmntyxn8+/6RKmjq0KXU5wSYNjHLAntn5t9vePYtvqgd4+Aqu74QjebUOzVxEFx0e62+/u97p7nbvX1dSU7rlHEelfj2zew7/+ditvv3IaN108KXQ5eSHpqKq9nD6dyCuADLA6ts2B3g41aAamxtanADtzG5nZJcA3gBvdvTVJwSIifbXn4HE+9P0nmTtxJJ+4uadJM0pH0h7HJuBPzGycmY0muuaxxt0PxtrMAHb38nhrgFozm2lmlcAyYEW8gZlNAx4C3uHuWxLWKyLSJ50Z54Pfe4KjbZ189W0LGFqRaMBoUUva47gbeJiop9ABVBE7dZQdJbWI03sgZ+XuHWZ2B7AKSAH3ufsGM7s9u3858EmiU2RfMzOADnevS1i3iEgiX3+kidXPtvKFN13C7AkjQ5eTV5LeOb4i+6V+W3bTd9z927Em1xOdplqV4JgriUZjxbctjy3/BfAXSeoUETkXa7bt5f/8spGll53Pm+s0kDNXX+4cv5ez3Evh7quIhuaKiBSk/Ufb+MD965kyZhj/8MaLyJ7pkJjif6q6iEgvuTt/++BTtBw+wQ/few0jh5bOVOlJlPZdLCIiMd/63Xb+a+OLfGTJXC6ZMjp0OXlLwSEiAvzxhQP8439u4jVzJ/CeRTNDl5PXFBwiUvIOn+jgzvvXM2Z4BV9886W6rtEDXeMQkZL3yR//ke2tR/ju/7yKscMrQ5eT99TjEJGS9sPHm3lo3Qu8/zW1XHXBuNDlFAQFh4iUrGdbDvOJH/+RK2eO5c5X14Yup2AoOESkJB1v7+TO765nSHkZdy9bQKoEHwHbV7rGISIl6XMrN7Fx10Hue3cdE0eV5iNg+0o9DhEpOas27Ob//W4771k0k1fPPS90OQVHwSEiJeWF/cf48INPcfHkUXx4yYWhyylICg4RKRkdnRk+cP96OjPOV966gCHlmiq9L3SNQ0RKxl2/bGTt9n3cvewyZowfHrqcgqUeh4iUhEeb0tzzSBNvqZvC0ssmhy6noCk4RKTotRw6wQe/9wSzakbwqTe8LHQ5BU+nqkSkqGUyzod+8CQHjrXzH+9ZSFWlvvbOlXocIlLU/m/9Vn67pYVP3jyfuROrQ5dTFBQcIlK01j+/jy+u2syNF03k7VdOC11O0VBwiEhROnCsnTvvX8951UP5pzddoqnS+5FO9olI0XF3Pv7Q0+w6cJwf3H41o4bpEbD9ST0OESk69/9hB//59C7+5rUX8vJpY0KXU3SCB4eZLTGzzWbWZGYf7Wb/XDP7nZmdMLO/CVGjiBSOZ3Yf5NM/2cDi2vH85XUXhC6nKAU9VWVmKeAe4AagGVhjZivcfWOs2V7g/cAbA5QoIgXkaFsHd3x3PSOHVvClt1xGmaZKHxChexwLgSZ33+rubcADwNJ4A3ff4+5rgPYQBYpI4fjMTzbybMth7rr1MmpGDgldTtEKHRyTgR2x9ebsNhGRRFY8uZMH1uzgr145i0W140OXU9RCB0d3/Ujv04HMbjOztWa2tqWl5RzLEpFCsr31CB9/6Gkunz6GD14/J3Q5RS90cDQDU2PrU4CdfTmQu9/r7nXuXldTU9MvxYlI/mvryHDn/espM7h72WVUpEJ/rRW/0J/wGqDWzGaaWSWwDFgRuCYRKSBf+PkzPNV8gC/ccglTxlSFLqckBB1V5e4dZnYHsApIAfe5+wYzuz27f7mZTQTWAtVAxsw+CMx394PBCheR4DIZ5weP7+AbDc/xzquns+SiSaFLKhnB7xx395XAypxty2PLu4lOYYmIkMk4P/vjbu7+1Ra2vHiYBdNG8/Gb5oUuq6QEDw4Rkd7IDYzZE0bw5bcu4PUXTyKl+zUGlYJDRPKaAiP/KDhEJC8pMPKXgkNE8ooCI/8pOEQkLygwCoeCQ0SCyg2MWTXDFRh5TsEhIkF0Fxh3L7uMmy85X4GR5xQcIjKoFBiFT8EhIoNCgVE8FBwiMqAUGMVHwSEiA0KBUbwUHCLSrxQYxU/BISL9QoFROhQcInJOFBilR8EhIn2iwChdCg4R6ZXOjLO99Qibdh1i066D/NfGF9n84iEFRglScIjIGQ6f6OCZXQfZtOsgG7NBsXn3IY61dwKQKjPmTRqpwChRCg6REubuNO87xqZdB0/2JDbuOsjze4+ebFM9tJx5k6q59YqpzD+/mvmTqpk9YQRDK1IBK5eQFBwiJeJ4eyebdx/KhkQ2KHYf5NDxDgDMYMa44Vw0uZo3Xz6FeZOqmXd+NeePGoqZehRyioJDpMi4O3sOnWBjPCB2HWRry2EyHrWpqkwxd+JI3nDp+cybVM3886u58LyRDB+irwTpmf6WiBSwto4MTXsOn+pF7I6CYu+RtpNtJo8exrxJ1dx00cSoFzGpmmljqyjTdQnpIwWHSJ463t7JwWPtHMj+7D8a/W49coJndh9i065DNO05RHtn1I2oLC/jwvNGcv28CScDYt7EakZVVQT+k0ixCR4cZrYEuBtIAd9w93/K2W/Z/TcBR4F3u/u6QS9UpA86OjMnv/gPHGtn/7H2k2HQFQRdyye3H2vjwLF2jrdnznrcmpFDmD+pmlfMqWHepJHMn1TNzPHDKU+VDeKfTkpV0OAwsxRwD3AD0AysMbMV7r4x1uxGoDb7cyXw9exvkQHh7nRmnPZOp60jQ1tnhvbODG0dGQ4d7zj5xX7GF34sCLp+Dp/oeMn3Gl6ZYtSwCqqHVTC6qoIZ46sYNWwUo6sqT20fVsGo7M/oqoqT+0RCCd3jWAg0uftWADN7AFgKxINjKfAtd3fg92Y22swmufuuwS+3NLk7GY9uAMt41w/R78yZy53ZZe9ads9+GXPq9dnlzuy++PG9m+WOTIa2Tqc954u8rTNDe4fT1tl5+hd9R7ZNZ4a2Du9m26nl9g4/7XhtnRnce//5VJaXnfblfv7oocydNDL6oh9Wyahh5acHQVXUrnpoBZXl6iFI4QkdHJOBHbH1Zs7sTXTXZjLQ78Hxmy0tfPanUWZ57JvjjO8Q73bxrK/x09p7t9u7X49au596XbTc1dZPrp/RPrvctdO7OZ7H9hHb1tUWODkKJ9+lyoyKlFGZKqOyvIyK+O9UGRXlZVSmjCEVZYwYWn5yf2WqLHpdrG3X8qk2p/aPHHr6v/5HDavQ/QxSckIHR3fDOnK/qnrTBjO7DbgNYNq0aX0qZsSQci48b2S375xbRHxcu522vfvXnK197oEttsEs2h39tpPHjn7bGftPf018PXdftN9i72/dHK9rvazMKDOjzE4tpyzaX2ZGqizaZ7nLZpSVkX1t9nVl0b4zlnOOn7tcXhb7oi+3WBhEX+i6c1lk8IQOjmZgamx9CrCzD21w93uBewHq6ur69O/ky6eP4fLpY/ryUhGRkhH6BOsaoNbMZppZJbAMWJHTZgXwTotcBRzQ9Q0RkXCC9jjcvcPM7gBWEQ3Hvc/dN5jZ7dn9y4GVRENxm4iG4/5ZqHpFRCT8qSrcfSVROMS3LY8tO/C+wa5LRES6F/pUlYiIFBgFh4iIJKLgEBGRRBQcIiKSiIJDREQSMU8yKU+BMLMWYHsfXz4eSPdjOYVOn8fp9Hmcos/idMXweUx395qeGhVlcJwLM1vr7nWh68gX+jxOp8/jFH0Wpyulz0OnqkREJBEFh4iIJKLgONO9oQvIM/o8TqfP4xR9Fqcrmc9D1zhERCQR9ThERCQRBUeMmS0xs81m1mRmHw1dT0hmNtXM/tvMNpnZBjP7QOiaQjOzlJmtN7Ofhq4ltOwjnB80s2eyf0euDl1TKGb219n/R/5oZveb2dDQNQ00BUeWmaWAe4AbgfnAW81sftiqguoAPuTu84CrgPeV+OcB8AFgU+gi8sTdwM/dfS5wKSX6uZjZZOD9QJ27X0T0eIhlYasaeAqOUxYCTe6+1d3bgAeApYFrCsbdd7n7uuzyIaIvhslhqwrHzKYArwe+EbqW0MysGrgO+CaAu7e5+/6wVQVVDgwzs3Kgim6eUFpsFBynTAZ2xNabKeEvyjgzmwEsAB4LW0lQdwEfBjKhC8kDFwAtwL9lT919w8yGhy4qBHd/Afhn4HlgF9ETSn8RtqqBp+A4xbrZVvJDzsxsBPBD4IPufjB0PSGY2c3AHnd/PHQteaIceDnwdXdfABx4bNd9AAACk0lEQVQBSvKaoJmNITozMRM4HxhuZv8jbFUDT8FxSjMwNbY+hRLocr4UM6sgCo3vuPtDoesJ6FrgDWa2jegU5qvN7NthSwqqGWh2964e6INEQVKKrgeec/cWd28HHgKuCVzTgFNwnLIGqDWzmWZWSXSBa0XgmoIxMyM6h73J3b8Uup6Q3P1j7j7F3WcQ/b34tbsX/b8qz8bddwM7zOzC7KbXABsDlhTS88BVZlaV/X/mNZTAQIHgzxzPF+7eYWZ3AKuIRkbc5+4bApcV0rXAO4CnzeyJ7LaPZ58RL3In8J3sP7K2An8WuJ4g3P0xM3sQWEc0EnE9JXAHue4cFxGRRHSqSkREElFwiIhIIgoOERFJRMEhIiKJKDhERCQRBYeIiCSi4BARkUQUHCIikoiCQ0REElFwiAwgM3vYzNzM7uxm32ez+0r+GR9SWDTliMgAMrOxRPMXnQdc7e7rs9tfA/wCeAa4wt2PhqtSJBkFh8gAM7NrgN8AzxFNP14FPAmMIgqNUp5MUwqQTlWJDDB3Xw18AqgF/hX4NjAReL9CQwqRehwigyD7rIafAa/Lbrrf3d8WsCSRPlOPQ2QQePQvtB/FNt0VqhaRc6Ueh8ggMLNaoof9tBNd29gALHT340ELE+kD9ThEBpiZDQG+BwwnevTs54CLUa9DCpSCQ2Tg/TOwAPiCu/8C+HvgUeAvzewtQSsT6QOdqhIZQGb2RqJrG48Bi9y9I7t9KvAEUA4scPet4aoUSUbBITJAzGwaUTiUEYXDczn7lwIPA2uIQqVt8KsUSU7BISIiiegah4iIJKLgEBGRRBQcIiKSiIJDREQSUXCIiEgiCg4REUlEwSEiIokoOEREJBEFh4iIJKLgEBGRRP4/UwbNVXr//WcAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(range(10), softmax(range(10)))\n",
    "plt.xlabel('x', fontsize = 20)\n",
    "plt.ylabel('softmax', fontsize = 20);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "\n",
    "That is, prior to applying softmax, some vector components could be negative, or greater than one; and might not sum to 1; but after applying softmax, each component will be in the interval (0,1), and the components will add up to 1, so that they can be interpreted as probabilities. \n",
    "\n",
    "Furthermore, the larger input components will correspond to larger probabilities. \n",
    "\n",
    "Softmax is often used in neural networks, to map the non-normalized output of a network to a probability distribution over predicted output classes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Activation Function\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-06-20T12:42:59.697609Z",
     "start_time": "2019-06-20T12:42:59.691063Z"
    },
    "code_folding": [],
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6224593312018546"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def sigmoid(x):\n",
    "    return 1/(1 + np.exp(-x))\n",
    "\n",
    "sigmoid(0.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-06-20T12:44:06.636552Z",
     "start_time": "2019-06-20T12:44:06.465463Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAETCAYAAAAh/OHhAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3Xl4XPV97/H3V6ttWXiRvG8yeGM1GGESCImJWWwKOCQ0hdCEJdSXFrLQ2zSkvUlzL20SQpMb0kC4LgGyAn0aHCvE7EtIQgAbI2/YxsZ4keVN3i1hWaP53j/OyIzFyNKRpTmzfF7PM8+c5Tejr89I8/HZfj9zd0RERLqqIOoCREQkuyg4REQkFAWHiIiEouAQEZFQFBwiIhKKgkNEREJRcIiISCgKDhERCUXBISIioRRFXUBvqKys9KqqqqjLEBHJKm+88UaDuw/prF1OBkdVVRWLFy+OugwRkaxiZhu70k6HqkREJBQFh4iIhKLgEBGRUBQcIiISioJDRERCiTQ4zOxBM9thZis6WG9m9kMzW2dmy8xsWrprFBGRo0W9x/EwMOsY62cDExOPucCP01CTiIgcQ6T3cbj7y2ZWdYwmc4CfeTC+7atmNtDMRrj71rQUKCJ5w91pjsVpjsWJtcaJxZ2W1jixVicWD+ZjrYll7da1tPqR6bg78TjE3fHE+8Y9mI8HC46aD9Y77gTzBNPJdbXNJ16emD56eaIx1VWD+eikTu/hOy6ZfgPgKGBz0nxdYtkHgsPM5hLslTB27Ni0FCcimeNgc4zNu5vY29TCweYYjc0xDiSeG5tjHDiUmD78/nTQrpWDienWuHf+gzLcLR87Ke+Dw1IsS/nJuvs8YB5AdXV19n/6InIUd2dvUwsbdjWyaXcTGxqa2Li7kY27mti4q5GGg4c7fG2BQVlpEf0Tj7bpoeV9EtOF9O9TRL+SIkqLCiguLKCo0CguKKCwwILpwgKKCuzIuqKCAooLjaKk5YUFRlGBUWCGGZhBgVniAbSbt0S7I/ME8xC81hJfgcF023JLmg7m0y3Tg6MOGJM0Pxqoj6gWEell7s7OA81s2NUUBETiuS0c9h+KHdV+5IA+jK3ox0UnD2NcRRljB/djcFlJIhyCMOhfWkTf4sJIvmBzVaYHRw1wm5k9CpwL7NP5DZHc4u4s3riHx5dsYeHyrex7r+XIusICY/SgvoyrKOPMMQMZV9GPqooyxlX0Y8zgfvQpLoyw8vwVaXCY2SPADKDSzOqAfwGKAdz9fmAhcBmwDmgCboymUhHpae82NDJ/SR3za7ewefd79C0u5JJTh3H2uEGMqyhj3OB+jBrUl+LCqC/+lPaivqrq2k7WO3BrmsoRkV62u/EwTyyr5/ElW6jdvBczOP+kSr48cxKXnjac/qWZfhBEIPMPVYlIlmuOtfLCqh38eskWXlqzg1jcmTK8nK/NnsKcM0cxfECfqEuUkBQcItLj3J1FG/Yw/80t/G5ZPfsPxRhaXsqN51dx1VmjOWXkCVGXKMdBwSEiPWZDQyOPtztvMeu04Vx11ijOn1BJYYGubMoFCg4R6RH/tXgzX3t8Oe7O+RMquf2iSVx66nDKdN4i5+gTFZHj4u788Pl1/N/n3uaCiZXcffVUnbfIcQoOEem2WGucry9YwSOvb+aT00Zx16fO0OWzeUDBISLd0nQ4xm2/epMXVu/gtgsn8D8vmaS7s/OEgkNEQms42MxNDy9ixZZ9/NtVp3HdueOiLknSSMEhIqG829DI9Q++zo4Dh5j32WouOmVY1CVJmik4RKTL3ty0h8//dDEAj/zNhzhr7KCIK5IoKDhEpEuefWs7X3hkCcNO6MPDN05nfGVZ1CVJRBQcItKpX7y6kW8sWMFpowbw4A3nUNm/NOqSJEIKDhHpkLvzvWfe5kcvruPCyUO497pp9CvR10a+02+AiKTU0hrnjl8v59dL6rjmnDH86ydOo0j3aAgKDhFJ4WBzjL/9xRv8YW0Dt180iS/OnKB7NOQIBYeIHGXH/kPc8NAi1mw/wHevPoNPV4/p/EWSVxQcInLEuh0HuP7BRexpOsxPrq9mxuShUZckGUjBISJAsKdx9f1/pqiggMfmfpjTRw+IuiTJUAoOEQHgB8+v5eChGE99+QImDC2PuhzJYLpEQkR4Z+dBHlu0mevOHavQkE4pOESEu59aQ5+iAr4wc2LUpUgWUHCI5Lk3Nu7hqZXbmPvRk3RHuHSJgkMkj7k7dz25msr+pdx8wfioy5EsoeAQyWPPr9rB6xt286WLJmpscOkyBYdInmqNO3c9tZrxlWVcc45u8pOuU3CI5Klfv1HH2h0H+cqlkzVOuISi3xaRPHSopZXvP/s2U8cMZPZpw6MuR7KMgkMkDz30pw1s23+Ir82eos4LJTQFh0ie2dt0mPteWsfHpwzlQydWRF2OZCEFh0ieuffFdRxsjvGPsyZHXYpkqciDw8xmmdkaM1tnZnekWD/AzH5rZkvNbKWZ3RhFnSK5oG5PEz99ZSOfmjaaKcNPiLocyVKRBoeZFQL3ArOBU4BrzeyUds1uBd5y96nADOB7ZlaS1kJFcsT3n30bDG6/eFLUpUgWi3qPYzqwzt3Xu/th4FFgTrs2DpRbcAavP7AbiKW3TJHst2rrfua/uYUbz6ti1MC+UZcjWSzq4BgFbE6ar0ssS/Yj4GSgHlgOfMnd4+kpTyR33PXUaspLi/jbGSdFXYpkuaiDI9V1gN5u/lKgFhgJnAn8yMw+cHDWzOaa2WIzW7xz586er1Qki73yTgMvrdnJrRdOYGA/HemV4xN1cNQByX0djCbYs0h2I/C4B9YB7wJT2r+Ru89z92p3rx4yZEivFSySbdo6Mhw5oA/Xn1cVdTmSA6IOjkXARDMbnzjhfQ1Q067NJmAmgJkNAyYD69NapUgWW7h8G0vr9nH7xZPoU1wYdTmSAyLtDtPdY2Z2G/A0UAg86O4rzeyWxPr7gTuBh81sOcGhra+6e0NkRYtkkZbWOHc/vZrJw8r55LTRUZcjOSLyfpTdfSGwsN2y+5Om64FL0l2XSC549PVNbNjVxIM3VFNYoK5FpGdEfahKRHrJweYY9zy/lnPHD+bCyUOjLkdyiIJDJEc98If1NBw8zB3qyFB6mIJDJAftPNDMvJfXc9npwzlr7KCoy5Eco+AQyUE/fH4tzbE4/3CJOjKUnqfgEMkx7zY08sjrm7h2+hhOHNI/6nIkByk4RHLMvz+9hpKiAr44c2LUpUiOUnCI5JDazXv53fKt3HzBiQwt7xN1OZKjFBwiOcLd+c6Tq6goK2HuR0+MuhzJYQoOkRyxrG4fr67fza0XTqB/aeT39koOU3CI5IgFtfWUFBbwqbPVtYj0LgWHSA5ojTtPLKtnxuQhDOhbHHU5kuMUHCI54LX1u9hxoJk5Z7YfB02k5yk4RHLAgtp6ykoKmXmy+qSS3qfgEMlyzbFWFq7YyqWnDdd4G5IWCg6RLPf7NTs5cCimw1SSNgoOkSy3YGk9FWUlnH9SRdSlSJ5QcIhksYPNMZ57azt/ccYIigr15yzpod80kSz2zMptNMfizDlzZNSlSB5RcIhksQW19Ywe1JdpGnND0kjBIZKldh1s5o/rGrhi6kiN8CdpdcwObcxsfTff1939pG6+VkS6YOHyrbTGXYepJO066wmtAPB2y0qAEYnpGLALqEh6r63A4Z4qUERSW1Bbz+Rh5UwZfkLUpUieOeahKnevcvfxbQ9gKrAFeBW4EOjj7iOAPsDHgdeAOuCM3i1bJL/V7Wli8cY9XKm9DYlA2HMc/wYMBGa4++/dPQ7g7nF3f4kgTAYn2olIL/nt0q0AXDlVwSHpFzY4rgIWuHvKQ1HufghYAHzyeAsTkY4tqN3CtLEDGTO4X9SlSB4KGxwVQGd9Nhcn2olIL1iz7QCrtx1QFyMSmbDB8Q5wtZkNSLXSzAYBVwPdvRpLRDpRs3QLhQXGZaeP6LyxSC8IGxz3AyOB183sc2ZWZWZ9E8/XE5wcHw7c29OFikgwrnjN0nrOn1DJkPLSqMuRPBVqYGJ3/5GZTQS+ADyUookB/+Hu9/VEcSJytDc372Xz7vf40sxJUZcieSz0iPbu/iUzexS4CTgLGADsA5YAD7v7Kz1booi0qamtp6SogEtPHRZ1KZLHQgcHgLv/GfhzD9ciIscQa43zxLJ6Ljp5KOV9NK64RCfyvqrMbJaZrTGzdWZ2RwdtZphZrZmtNLPfp7tGkUzwyju7aDh4mCun6moqiVZnfVWNTUxucffWpPlOufumztqYWSHBifSLCe44X2RmNe7+VlKbgcB9wCx332RmGlRZ8lLN0nrKS4uYMXlI1KVInuvsUNUGgr6qTgbeTprvjHfhvQGmA+vcfT1A4tzJHOCtpDafAR5vCyJ339GF9xXJKYdaWnlqxTZma1xxyQCdfbn/jCAE9rWb7ymjgM1J83XAue3aTAKKzewloBy4x91/1v6NzGwuMBdg7Ngu7xiJZIUXV+/gYLPGFZfMcMzgcPcbjjXfA1INItA+mIqAs4GZQF/gz2b2qru/3a62ecA8gOrq6p4MN5HILaitp7J/KR/WuOKSAaI+OV4HjEmaHw3Up2jzlLs3unsD8DJBL70ieWH/oRZeWLODK6aOoLBAAzZJ9LodHGY22syuMLPPmtmVZja6G2+zCJhoZuPNrAS4Bqhp12YBcIGZFZlZP4JDWau6W7dItnl6xTYOx+LqCVcyRuj7OBJXVs0juBKq/bpngVvcfUNX3svdY2Z2G/A0UAg86O4rzeyWxPr73X2VmT0FLAPiwAPuviJs3SLZqmZpPWMH9+PMMQOjLkUECBkcZjYc+BPBSe0NBIeNthKMCPgR4BLgj2ZW7e7buvKe7r4QWNhu2f3t5u8G7g5Tq0gu2HHgEH9a18CtF07QuOKSMcLucXydIDS+Cnzf3VvbViTuybgd+C7wv4DbeqpIkXz1u2VbiTsaV1wySthzHH8BPOPudyeHBoC7t7r7vwPPAJf3VIEi+axmaT0njziBCUPLoy5F5IiwwTEceKOTNm8k2onIcdi0q4k3N+3V3oZknLDBsQ8Y10mbsbx/w6CIdFPN0i0AXKGrqSTDhA2OPxKMAHheqpVmdi7wl4l2ItJN7s6C2nqmVw1m1MC+UZcjcpSwJ8f/jeA8x+8T/Uq9SHBV1XBgBnAtwSWz3+rBGkXyzuptB1i74yD/+onToi5F5APCjgC4xMyuBh4GriPogLCNAbuBm9y9s/MgInIMC2rrKdK44pKhujMC4BNmNo6gF9tpvD8C4JvAb9y9sWdLFMkv8bjz26X1XDCxksFlJVGXI/IB3R0BsBH4VeIhIj3ojU172LL3Pb5y6eSoSxFJKepODkWknZraevoUF3DxKRpXXDJTt/Y4zOwMgh5qRwOpBj92d7/zeAoTyUctrXF+t3wrF508jLLSbv15ivS6sH1VDQZ+DsxqW9RBUwcUHCIh/XFdA7sbD2vAJsloYf9L8wNgNvAc8AtgCxDr6aJE8lVNbT0D+hbzsUkaV1wyV9jguBx4xd0v6Y1iRPLZe4dbeXrlNuacOZKSIp1+lMwV9rezEHilNwoRyXfPr95O0+FWdTEiGS9scCwBTuyNQkTy3YLaeoadUMq54zWuuGS2sMFxJ3C5mX2kN4oRyVf7mlp4ac0OrjhjpMYVl4wXtsuRF8zsGmC+mT1BsAeSsidcd/9ZD9QnkheeXLGVllbX1VSSFcJejltC0NXIIOD6xMPbN0ssU3CIdFHN0npOrCzjtFEnRF2KSKfCXlX1bYKweAt4DKhHl+OKHJft+w/x5/W7+OLHJ2pccckKYYPjGmA5cI67H+6FekTyzm+X1uMOV2qkP8kSYU+ODyQYc1yhIdJDapbWc/qoAZw0pH/UpYh0SdjgWAVogACRHvJuQyPL6vZpXHHJKmGD43vAJ8xsUm8UI5JvamrrMYPLz1BwSPYIe45jC/AU8JqZ3QO8QceX4758nLWJ5DR3Z8HSLZw7fjDDB/SJuhyRLgsbHC8RXGprwDf44KW4yQq7WZNIXlhZv5/1Oxv5mwvUGYNkl7DB8X84dliISBctqN1CcaEx+7ThUZciEkrYO8e/2Ut1iOSVYFzxrXxs0lAG9tO44pJd1HezSARe37CbbfsP6d4NyUoKDpEILKitp19JIRedPDTqUkRCC9tX1QtdaBYH9hPc8zHf3Rd38p6zgHsITqY/4O7f6aDdOcCrwF+5+3+HqVskkxyOxVm4fCuXnDKMfiUaV1yyT9jf2hmJ57Yrq9pLXv4J4A4zu9/db031ZmZWCNwLXAzUAYvMrMbd30rR7i7g6ZD1imScP6zdyb73WtQTrmStsIeq+gC/Ad4G/hqoAvomnj+bWP4bYDRwKVAL3GJmN3XwftOBde6+PtGNyaMEve+29wXg18COkPWKZJwFtfUM6lfMRyZWRl2KSLeEDY6vA9XAue7+K3ff5O7NiedfAh8CzgFucfdngUuAPcDnO3i/UcDmpPm6xLIjzGwUcBVwf8haRTJOY3OMZ9/azmWnj6C4UKcYJTuF/c29Dnjc3fenWunu+wj2DP46Mb8LeBI4tYP36+hwV7IfAF9199ZjFWZmc81ssZkt3rlz57GaikTmuVXbea+lVYepJKuFPccxEmjppE0LR3eEWEdwiCuVOmBM0vxogjE+klUDjybGKagELjOzmLv/JrmRu88D5gFUV1frJkXJSAtq6xk5oA/V4wZFXYpIt4Xd49gCXGFmKQPHzIqBKzn6y38IsLeD91sETDSz8YnRBa8BapIbuPt4d69y9yrgv4G/ax8aItlgT+NhXn57J1dMHUmBxhWXLBY2OH4OTAKeNbPzzawAwMwKzOwjwLPAhES7NucBK1O9mbvHgNsIrpZaBfyXu680s1vM7JaQtYlktIUrthKLu276k6wX9lDVtwgOHV0GvAzEzWw3MJgghIyg99xvAZjZCGApwXmPlNx9IbCw3bKUJ8Ld/YaQ9YpkjAW19UwY2p9TRmhcccluofY43P2wu19OMO74iwRdqg8muOHvReAGd7+sbYRAd9/q7tfqhj3Jd/V73+P1d3czZ+pIjSsuWa9bt626+885+nCUiBzDE8uC0346TCW5QBeSi6TBgtp6po4ZyLiKsqhLETluCg6RXrZuxwFW1u9nzlTtbUhuOOahKjOLE3RaeIq7v52Y78o9Eu7u6r1NhGBc8QKDy88Y0XljkSzQ2Zf7ywRB0dRuXkS6IBhXvJ7zTqpk6AkaV1xywzGDw91nHGteRI5tWd0+Nu5q4tYZE6IuRaTH9MjhJDOrBC4g2DN5rrN+pUTyxYLaekoKC7hU44pLDgl1ctzM/tbMXjOzwUnLzia46/u/CW7ke8XMdOmI5L3WuPPbZfVcOGUIA/oWR12OSI8Je1XVXxGc+N6dtOxuYBDwEEFwnAOouxDJe6+u38XOA83qCVdyTtjgmAgsa5tJHKL6GPATd7/Z3a8g6LjwMz1Xokh2qqmtp39pER+fonHFJbeEDY4Kjh6F7/zE8/ykZX8Axh1PUSLZrjnWysIVW7nk1GH0KS6MuhyRHhU2OHYTjInR5mME93m8krTM6Xj8DZG88NKanRw4FNNhKslJYYNjFcF4HBVmNpDgnMeidiMCVgHbeqg+kaxUU1tPRVkJ559UEXUpIj0ubHDcQzC6Xx3BWOHDgfvaVppZIfARgq7URfLSgUMtPLdqO5efMYIijSsuOSjUfRzuXpMYYGluYtEv3f0XSU0uIjhM9XQP1SeSdZ59azvNsbh6wpWcFfoGwOSxvVOse5rg0lyRvLWgtp7Rg/oybaz+FCQ3aT9apAc1HGzmj+sauFIDNkkOU3CI9KCFy7fSGnddTSU5TcEh0oNqauuZPKycycPLoy5FpNcoOER6yDs7D7J44x6dFJecp+AQ6SHfe2YNZSWFfLp6TNSliPQqBYdID3hz0x4WLt/G33z0RIaUl0ZdjkivUnCIHCd359tPrqayfwk3X3Bi1OWI9DoFh8hxenHNDl5/dzdfmjmR/qU9MjaaSEZTcIgch9a4c9eTa6iq6Mc108dGXY5IWig4RI7D40vqWLP9AF+5dArF6pdK8oR+00W66VBLK99/9m2mjh7AZadrTHHJHwoOkW766Ssb2LrvEHfMPlndi0heUXCIdMO+phbufXEdF04ewoc15obkGQWHSDfc99I6DjTH+MdZU6IuRSTtIg8OM5tlZmvMbJ2Z3ZFi/XVmtizxeMXMpkZRp0ib+r3v8dArG/jkWaM5ecQJUZcjknaRBkdixMB7gdnAKcC1ZnZKu2bvAh9z9zOAO+lgLBCRdPn+s28D8PeXTIq4EpFoRL3HMR1Y5+7r3f0w8CgwJ7mBu7/i7nsSs68Co9Nco8gRq7ft59dL6rjhvCpGDewbdTkikYg6OEYRjF3epi6xrCOfB55MtcLM5prZYjNbvHPnzh4sUeR9331qDeWlRfzdjJOiLkUkMlEHR6prGD1lQ7MLCYLjq6nWu/s8d6929+ohQ4b0YIkigVfX7+KF1Tv4uwsnMLBfSdTliEQm6o516oDkPqhHA/XtG5nZGcADwGx335Wm2kSOaOvIcMSAPtxwXlXU5YhEKuo9jkXARDMbb2YlwDVATXIDMxsLPA581t3fjqBGEZ5csY2lm/dy+8WT6FNcGHU5IpGKdI/D3WNmdhvwNFAIPOjuK83slsT6+4FvABXAfYm7c2PuXh1VzZJ/Wlrj3P30GiYN68+npunaDJGoD1Xh7guBhe2W3Z80fTNwc7rrEmnz6KLNvNvQyE+ur6awQF2LiER9qEokozU2x7jnubVMrxrMx6cMjbockYyg4BA5hgf+8C4NB5u547Ip6shQJEHBIdKBhoPNzHv5HWadOpxpYwdFXY5IxlBwiHTgP55fy6FYnK/Mmhx1KSIZRcEhksLGXY388rVN/NU5YzhpSP+oyxHJKAoOkRTufnoNxYUFfHnmxKhLEck4Cg6RdpbV7eWJZVu5+YLxDD2hT9TliGQcBYdIEnfnO0+uZnBZCXM/emLU5YhkJAWHSJKX1zbwyju7+MLHJ1DepzjqckQykoJDJKE1HuxtjBncl+vOHRd1OSIZS8EhArx3uJVbfvEGq7bu56uzplBSpD8NkY5E3leVSNR2Nx7m8z9dRO3mvfzvK0/l8jNGRl2SSEZTcEhe27Sriesfep36ve/x4+umMeu0EVGXJJLxFBySt5bV7eWmhxcRizu/vPlcqqsGR12SSFZQcEheenHNDm795RIGl5Xw8I3TmTBUd4eLdJWCQ/LOY4s28U/zVzBleDkP3XgOQ8t1k59IGAoOyRvuzg+eW8s9z6/lo5OGcN910+hfqj8BkbD0VyN5oaU1zv+av4LHFm/m6rNH8+1Pnk5xoS65FekOBYfkvMbmGLf+agkvrdnJFz8+gdsvnqRBmUSOg4JDctrOA83c9PAiVtbv41tXnc5nzh0bdUkiWU/BITlr/c6DXP/Q6zQcOMx/fq6amScPi7okkZyg4JCc9MbGPdz800UUmPHI3A9x5piBUZckkjMUHJJznlm5jS888iYjBvTh4RunU1VZFnVJIjlFwSE5w935+asb+WbNSk4fPZAHr6+mon9p1GWJ5BwFh2S9dxsamb+kjvm1W9i8+z1mThnKf3zmLPqV6NdbpDfoL0uy0u7GwzyxrJ7Hl2yhdvNezOD8kyq5/aJJXDl1JEW6R0Ok1yg4JGscamnlhdU7eHzJFl5as4NY3JkyvJyvzZ7CnDNHMXyAug4RSQcFh2S0eNxZvHEP89+s43fLtrL/UIyh5aXceH4VV501mlNGnhB1iSJ5R8EhGWn9zoPMf3ML89/cQt2e9+hbXMis04Zz1VmjOH9CJYUFuvNbJCoKDolcPO5sP3CIjbuaeKt+PwuW1rN0814KDM6fUMnfXzyJS08dTpk6JBTJCJH/JZrZLOAeoBB4wN2/0269JdZfBjQBN7j7krQXKscl1hqnfu8hNu5uZMOuJjY2NLJxdxMbdzWycVcTzbH4kbZThpfzT5cF5y2GnaDzFiKZJtLgMLNC4F7gYqAOWGRmNe7+VlKz2cDExONc4MeJZ8kQ7k7T4VYam2Pse6+FzXua2NAQhMKGXU1s2t3E5t1NxOJ+5DWlRQWMq+jHuIoyPjZpCGMryqiq6EdVRRljBveL8F8jIp2Jeo9jOrDO3dcDmNmjwBwgOTjmAD9zdwdeNbOBZjbC3bemv9zM5u64Q0s8TqzVibX6kemW1jixuBNrjdPS6sTibfOJZUfWxTnYHITAwcTjyPShGI2HYxxsbuXgoRYa29odjuH+wXrKS4sYV9mPU0acwOzThlNVUcbYRDgMLS+lQOcpRLJS1MExCticNF/HB/cmUrUZBfR4cPz+7Z3c+cT7meUpvg0/sMQ/ONv2umC6bbm/P530GndP2S7uwbq4O3GHuDskntvmg0VHz/e0ogKjf58iykqKKO9TRFlpEQP6FjN6YF/KSgvpX1pM/9JCykqL6N+niPI+xYwe1JeqijIG9StW9+UiOSjq4Ej1rdL+668rbTCzucBcgLFju9d1dv/SIiYPK+/0p7df1P7L0YC2RZa03pJebFi7Nu8vLygIXlNgUGBGQWJlQduyguC1RnKb4DVmUFxYQFGBUVRYQHGhUVRQQFGhvT+dWFdUaBS3W1dcWJAIhCAkSosK9OUvIkeJOjjqgDFJ86OB+m60wd3nAfMAqquru/V/77PHDeLscYO681IRkbwRdb8Mi4CJZjbezEqAa4Cadm1qgM9Z4EPAPp3fEBGJTqR7HO4eM7PbgKcJLsd90N1XmtktifX3AwsJLsVdR3A57o1R1SsiItEfqsLdFxKEQ/Ky+5OmHbg13XWJiEhqUR+qEhGRLKPgEBGRUBQcIiISioJDRERCUXCIiEgolqpbjWxnZjuBjd18eSXQ0IPl9LRMrw8yv0bVd3xU3/HJ5PrGufuQzhrlZHAcDzNb7O7VUdfRkUyvDzK/RtV3fFTf8cn0+rpCh6pERCQUBYeIiISi4PigeVEX0IlMrw8yv0bVd3xU3/HJ9Po6pXMcIiISivY4REQklLwMDjP7SzNbaWZxM6tut+5rZrbOzNaY2aUdvH6wmT1rZmsTz702iIeZPWavDmgCAAAF40lEQVRmtYnHBjOr7aDdBjNbnmi3uLfqSfFzv2lmW5JqvKyDdrMS23Sdmd2RxvruNrPVZrbMzOab2cAO2qV1+3W2PRLDCPwwsX6ZmU3r7ZqSfvYYM3vRzFYl/k6+lKLNDDPbl/S5fyNd9SXVcMzPLOJtODlp29Sa2X4z+3K7NpFvw24LxqnOrwdwMjAZeAmoTlp+CrAUKAXGA+8AhSle/13gjsT0HcBdaar7e8A3Oli3AaiMYFt+E/iHTtoUJrbliUBJYhufkqb6LgGKEtN3dfRZpXP7dWV7EAwl8CTBAJEfAl5L42c6ApiWmC4H3k5R3wzgiXT/voX5zKLchik+720E90hk1Dbs7iMv9zjcfZW7r0mxag7wqLs3u/u7BGOATO+g3U8T0z8FPtE7lb7PgvFbPw080ts/qxdMB9a5+3p3Pww8SrANe527P+PuscTsqwQjSEatK9tjDvAzD7wKDDSzEekozt23uvuSxPQBYBUwKh0/u4dFtg3bmQm84+7dvSk54+RlcBzDKGBz0nwdqf9ghnliFMLE89A01HYBsN3d13aw3oFnzOyNxPjr6XRb4lDAgx0ctuvqdu1tNxH8DzSVdG6/rmyPjNhmZlYFnAW8lmL1h81sqZk9aWanprWwQGefWUZsQ4KRTTv6D1/U27BbIh/IqbeY2XPA8BSr/tndF3T0shTLev2ysy7Wei3H3ts4393rzWwo8KyZrXb3l3u7PuDHwJ0E2+lOgsNpN7V/ixSv7bHt2pXtZ2b/DMSAX3bwNr22/VLoyvaI5HfxqALM+gO/Br7s7vvbrV5CcOjlYOK81m+Aiemsj84/s0zYhiXAlcDXUqzOhG3YLTkbHO5+UTdeVgeMSZofDdSnaLfdzEa4+9bEru+O7tTYprNazawI+CRw9jHeoz7xvMPM5hMcDumRL76ubksz+0/giRSrurpdu6UL2+964HJgpicOLqd4j17bfil0ZXv06jbrjJkVE4TGL9398fbrk4PE3Rea2X1mVunuaeuDqQufWaTbMGE2sMTdt7dfkQnbsLt0qOpoNcA1ZlZqZuMJ0v/1Dtpdn5i+HuhoD6anXASsdve6VCvNrMzMytumCU4Ir+jlmtp+dvIx46s6+LmLgIlmNj7xP7BrCLZhOuqbBXwVuNLdmzpok+7t15XtUQN8LnFl0IeAfW2HR3tb4nzaT4BV7v79DtoMT7TDzKYTfJfsSkd9iZ/Zlc8ssm2YpMMjBVFvw+MS9dn5KB4EX3B1QDOwHXg6ad0/E1zxsgaYnbT8ARJXYAEVwPPA2sTz4F6u92HglnbLRgILE9MnElyZsxRYSXCIJl3b8ufAcmAZwR/qiPb1JeYvI7g6550017eO4Dh3beJxfyZsv1TbA7il7XMmOMxyb2L9cpKu/ktDbR8hOKSzLGm7XdauvtsS22opwUUH56WrvmN9ZpmyDRM/vx9BEAxIWpYx2/B4HrpzXEREQtGhKhERCUXBISIioSg4REQkFAWHiIiEouAQEZFQFBwiIhKKgkNEREJRcIiISCgKDhERCUXBIdKLzOw3ZuZm9oUU6+5MrHsgitpEuktdjoj0IjMbDLwJDAM+7O5vJpbPBJ4BVgPneAcdMIpkIgWHSC8zs/OA3wPvAtMIOr9bCgwgCI2VEZYnEpoOVYn0Mnd/Bfg6QTf9/w/4BcHAU19UaEg20h6HSBokxl14Erg0segRd/9MhCWJdJv2OETSwIP/oc1PWvSDqGoROV7a4xBJAzObSDDGdAvBuY2VwHR3PxRpYSLdoD0OkV5mZqXAY0AZwTCx3wZOR3sdkqUUHCK979+Bs4DvuvszwL8AfwL+h5l9OtLKRLpBh6pEepGZfYLg3MZrwEfcPZZYPoZgLO8i4Cx3Xx9dlSLhKDhEeomZjSUIhwKCcHi33fo5wG+ARQShcjj9VYqEp+AQEZFQdI5DRERCUXCIiEgoCg4REQlFwSEiIqEoOEREJBQFh4iIhKLgEBGRUBQcIiISioJDRERCUXCIiEgo/x8qI07ye5N4ngAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(range(-10, 10), [sigmoid(i) for i in range(-10, 10)])\n",
    "plt.xlabel('x', fontsize = 20)\n",
    "plt.ylabel('sigmoid', fontsize = 20);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-06-21T03:12:03.482478Z",
     "start_time": "2019-06-21T03:12:03.475234Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Naive scalar relu implementation. \n",
    "# In the real world, most calculations are done on vectors\n",
    "def relu(x):\n",
    "    if x < 0:\n",
    "        return 0\n",
    "    else:\n",
    "        return x\n",
    "\n",
    "relu(0.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-06-21T03:12:06.248059Z",
     "start_time": "2019-06-21T03:12:06.092317Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAETCAYAAADZHBoWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAHxBJREFUeJzt3Xl41NW9x/H3l5Cw74RF9n1TIBgRcK1YVLRqVRS8tlrxUttHEJW6F7Xe3roXpbaWWm3r1QAKLrXu1qVuVCAJ+76GNewQCNnO/WPGacSEJJNkziyf1/PkYTK/3/zm85wh851zzm/Oz5xziIiIANTxHUBERKKHioKIiISoKIiISIiKgoiIhKgoiIhIiIqCiIiEqCiIiEiIioKIiISoKIiISEhd3wGqqnXr1q5r166+Y4iIxIwFCxbscs6lVmbfmCsKXbt2Zf78+b5jiIjEDDPbWNl9NXwkIiIhKgoiIhKioiAiIiEqCiIiEqKiICIiISoKIiISoqIgIiIhKgoiIlHuq3W7+fNn64nE5ZNVFEREotjevAImz8zixa82cqSwuNafT0VBRCRKOef4xSuL2JNXwFPj0miYUvuLUKgoiIhEqb99uZEPlu/gjgv6cmKHZhF5ThUFEZEotGzrAX791nLO6duG60/rGrHnVVEQEYkyhwuKmJixkOYNknn0ioGYWcSeO+ZWSRURiXcPvLGMdbvyeHH8qbRqXC+iz62egohIFPl79lZmzd/Mz8/uwYierSP+/CoKIiJRYvOew9w9dzFDOjdn8rm9vWRQURARiQKFxSVMzMgEgyfHppGc5OftWXMKIiJR4In3V5G1eR9PXz2ETi0besuhnoKIiGefrd7FM5+sZdzQTlw4sL3XLCoKIiIe7Tp0lFtmZ9EjtTFTLxrgO46Gj0REfCkpcUx5OZv9Rwp5YfxQGqQk+Y6knoKIiC/Pfb6ej1fm8ssL+9G3XVPfcQAVBRERLxbn7Ofhd1Ywqn9brhnWxXecEBUFEZEIO3Q0sIxF68b1eCTCy1hURHMKIiIRNvW1JWzac5iZE4bTvGGK7zjfop6CiEgEzV2Yw9zMLUwa2Yuh3Vr6jvMdKgoiIhGyflcev3xtCUO7tWTiOb18xymT96JgZreY2VIzW2JmGWZW33cmEZGaVlBUwqSMTOom1WHaVYNJqhM98wileS0KZtYBmASkO+dOBJKAsT4ziYjUhkfeWcHiLft59IqBnNC8ge845fLeUyAw2d3AzOoCDYGtnvOIiNSoj1bu5NnP1vPj4V0YNaCd7zjH5bUoOOe2AI8Bm4BtwH7n3HvH7mdmE8xsvpnNz83NjXRMEZGw7TyQz5TZ2fRt14S7R/fzHadCvoePWgCXAN2AE4BGZnbNsfs552Y459Kdc+mpqamRjikiEpaSEsets7PJKyjid1enUT/Z/zIWFfE9fHQusN45l+ucKwTmAiM8ZxIRqRF//HQdn63Zxf0/GEDPNk18x6kU30VhEzDMzBpa4Ct9I4HlnjOJiFRb5qa9PP7eSi4c2J6rTunkO06l+Z5TmAe8AiwEFgfzzPCZSUSkug7kFzIxI5O2Tevzvz88KaqWsaiI92UunHP3Aff5ziEiUhOcc9w9dzHb9ucz+6fDadYg2XekKvE9fCQiEldenp/Dm4u2cev3e3Nylxa+41SZioKISA1Zs/Mg972xlBE9WnHjWT18xwmLioKISA3ILyzmppcyaZCSxG+jeBmLinifUxARiQcPvb2CFdsP8vx1p9C2aewu4aaegohINb2/bAd/+WID40/vxvf6tvEdp1pUFEREqmHb/iP84pVsBpzQlNvP7+M7TrWpKIiIhKm4xDF5ZhYFRSVMH5dGvbrRv4xFRTSnICISpqc/WsO89Xt4fMwguqc29h2nRqinICIShq837GHaB6u4dPAJXDakg+84NUZFQUSkivYdLuDmjEw6tWzI/8TYMhYV0fCRiEgVOOe4c85idh48ytyfj6Bxvfh6G1VPQUSkCl6ct4l3lm7n9vP7MLBjc99xapyKgohIJa3cfpAH31zGmb1TueH07r7j1AoVBRGRSjhSUMzEjIU0qZ/M42MGUSdGl7GoSHwNhomI1JIH/7GMVTsO8cL4oaQ2qec7Tq1RT0FEpAJvL97GS/M28dOzunNGr/i+TryKgojIceTsPcwdcxYxqFNzpoyK/WUsKqKiICJSjqLiEm6emUWJg+lj00hOiv+3TM0piIiU48kPV7Ng416eHDuYzq0a+o4TEfFf9kREwvDF2l387qM1jDm5I5cMjp9lLCqioiAicow9eQXcMiuLbq0b8cAlA3zHiSgVBRGRUpxz/OLlbPbmFTJ9XBoNUxJrlF1FQUSklL98sYEPV+zkrtF9GXBCM99xIk5FQUQkaMmW/fzmrRWM7NuG60Z09R3HCxUFEREg72gRkzIyadEomUfHDIqr5bCrIrEGy0REynH/G0tZvzuPF284lZaNUnzH8UY9BRFJeK9nbeHlBTnc9L2ejOjR2nccr1QURCShbdydxz2vLuHkLi24eWQv33G8U1EQkYRVUFTCpIxM6hg8OXYwdRNgGYuKaE5BRBLW4++vJDtnP7//ryF0bJEYy1hURGVRRBLSp6ty+eMn6xg3tDOjT2rvO07UUFEQkYSTe/Aot87Opnfbxky9qL/vOFFFw0ciklBKShy3vZzNwfxCXrzhVBqkJPmOFFXUUxCRhPLsZ+v4dFUu917Unz7tmviOE3W8FwUza25mr5jZCjNbbmbDfWcSkfiUvXkfj7yzkvMGtOWaUzv7jhOVomH46EngHefcFWaWAugUABGpcQfzC5mYkUmbJvV4+PKBCbuMRUW8FgUzawqcCVwH4JwrAAp8ZhKR+OOc497XlpCz9zAzJwynecPEXcaiIr6Hj7oDucDzZpZpZs+aWSPPmUQkzsxZuIXXs7Zy88jeDO3W0necqOa7KNQFhgB/cM6lAXnAncfuZGYTzGy+mc3Pzc2NdEYRiWHrcg8x9fUlDO3WkpvO6ek7TtTzXRRygBzn3Lzg768QKBLf4pyb4ZxLd86lp6amRjSgiMSuo0XFTMzIJKVuHZ4cO5ikOppHqIjXouCc2w5sNrM+wbtGAss8RhKROPLw2ytZuvUAj1w+kPbNGviOExOi4eyjicCLwTOP1gE/8ZxHROLAP1fs4LnP13Pt8C6MGtDOd5yY4b0oOOeygHTfOUQkfuw4kM+UlxfRt10T7hrdz3ecmOJ7TkFEpEYVlzgmz8ziSEExv7s6jfrJWsaiKrz3FEREatIzn6zly3W7efjyk+jZRstYVJV6CiISNxZs3MMT76/iooHtuTK9k+84MUlFQUTiwv4jhUzKyKJ9s/r872UnaRmLMGn4SERinnOOu+cuZvuBfF6+cThN6yf7jhSz1FMQkZg38+vN/GPxNm4b1ZshnVv4jhPTVBREJKat3nGQB/6+lNN7tubGM3v4jhPzVBREJGblFwaWsWiUUpcnrhxEHS1jUW2aUxCRmPXrfyxnxfaDPP+TU2jTtL7vOHFBPQURiUnvLNnOC19t5IbTu/G9Pm18x4kbKgoiEnO27jvCHXMWcVKHZtx+fl/fceKKioKIxJSi4hImz8yiqLiEp8alkVJXb2M1SXMKIhJTpv9zDf/esIcnrhxEt9a6UGNNq3JRMLPOld3XObepqscXESnPV+t2M/2fq7ksrQOXDenoO05cCqensAFwldjPhXl8EZHv2JtXwC2zsujcsiG/uvRE33HiVjhv2n+j7KLQHBgMdAE+BjaGH0tE5D+cc9w+ZxG7Dh1l7s9Oo3E9fd6sLVVuWefcdeVtM7M6wC+BG4Frw48lIvIfL3y1kfeX7eDeC/txUsdmvuPEtRqdtnfOlTjnHiAwxPRQTR5bRBLT8m0H+J9/LOfsPqlcf1o333HiXm2dy/UFMKqWji0iCeJwQRETMzJp1iCZx8ZoGYtIqK2BuZaAzhUTkWr51d+XsTb3EC9cfyqtG9fzHSch1HhPwczOBa4CltT0sUUkcby5aCszv97MjWf14PRerX3HSRjhfE/hn8c5Vifgm+8x/CrcUCKS2DbvOcxdcxczuFNzbv1+b99xEko4w0dnl3O/A/YC7wKPOefKKx4iIuUqLC5h0sxMcDB9XBrJSVrGIpLCOSVVr5CI1JppH6wic9M+po9Lo1PLhr7jJBy9wYtI1PhizS5+//FarkrvxA8GneA7TkJSURCRqLD70FEmz8qie+tG3Hdxf99xElaFw0dm9uNwD+6c+1u4jxWRxOGcY8rL2ew7UshffjKUhilaxsKXyrT8X6jcAnilWfAxKgoiUqHnPt/ARytzeeDiAfQ/oanvOAmtMkXhJ7WeQkQS1pIt+3no7eWc268tPx7exXechFdhUXDO/TUSQUQk8eQdDSxj0apRPR69YiBmWsbCNw3ciYg3U19fysbdebz038No0SjFdxyhGkXBzFKBy4F+QCPn3A2l7u8GLHbOHamRlCISd17L3MKchTlMGtmLYd1b+Y4jQWEVBTMbDzwF1Oc/k8o3BDe3Bb4EJgB/roGMIhJnNu7O455XF3NK1xZMOqen7zhSSpW/p2Bm3wdmAKuAHwJ/KL3dObcEWApcWhMBRSS+FBSVMDEjk7pJdZg2No26WsYiqoTTU7gD2Aac5Zw7YGZpZeyzCBherWQiEpcee28li3L288w1J9OheQPfceQY4ZTodOBN59yB4+yTA7Sr7AHNLMnMMs3szTDyiEiM+GRVLjM+Xcc1wzpz/omVfouQCAqnKKQAeRXs0xworsIxbwaWh5FFRGLEzoP53DY7iz5tm3DvhVrGIlqFUxQ2ACdXsM+pwMrKHMzMOgIXAs+GkUVEYkBJieO22dkcOlrE9KvTqJ+c5DuSlCOcovA6cIaZjSlro5n9BBgIzKnk8aYBtwMlYWQRkRjwp3+t41+rdzH1ogH0btvEdxw5jnCKwiPAJiDDzGYRnFA2s5uCv88AVgPTKzqQmV0E7HTOLahgvwlmNt/M5ufm5oYRWUR8ydq8j0ffXckFJ7Zj3NBOvuNIBcy5qq51B2bWmcBid2eWsflfwNXOuS2VOM5vgB8BRQS+89AUmOucu6a8x6Snp7v58+dXObOIRN7B/EIufOozikscb006g2YNk31HSkhmtsA5l16ZfcO5RvOZwAHn3NlmNpBAT6EVsB/4qqJP/aU55+4C7goe92xgyvEKgojEDucc97y6hC37jjD7p8NUEGJEON9T+Aj4I/Bz59wiAt9JEBH5llcW5PBG9lamjOrNyV1a+o4jlRROUdgF1PiaRs65j4GPa/q4IhJ5a3MPMfX1pQzr3pKfna1lLGJJOBPNHwMjajiHiMSJo0XFTHwpk/rJdZh2VRpJdbQcdiwJpyjcC/QxswfNTIOEIvItD729gmXbDvDYmEG0a1bfdxyponCGj+4ClgB3A+PNLBvYzncv2emcc+OrmU9EYsiHy3fw/OcbuG5EV0b2a+s7joQhnKJwXanb7Sh/jSMHqCiIJIjt+/OZ8nI2/ds35a7RfX3HkTCFUxS61XgKEYlpxSWOybMyyS8sYfrVadSrq2UsYlWVi4JzbmNtBBGR2PWHj9fw1bo9PHLFQHqkNvYdR6pBV7cQkWpZsHEPv/1gNRcPOoExJ3f0HUeqSUVBRMK2/3AhkzKy6NC8Ab/+4YmY6fTTWBfWNZpFRJxz3Dl3ETsO5PPKz0bQpL7OUI8H6imISFgy/r2Zt5dsZ8p5fRjcqbnvOFJDVBREpMpW7TjIA39fyhm9WjPhjO6+40gNUlEQkSrJLyzmppcW0qR+XR6/chB1tIxFXNGcgohUyYNvLmPVjkP89fqhtGmiZSzijXoKIlJp7yzZxovzNjHhzO6c1TvVdxypBSoKIlIpW/Yd4fZXFjGoYzOmjOrjO47UEhUFEalQUXEJN2dkUuLgqXFppNTVW0e80pyCiFToqQ9XM3/jXqZdNZgurRr5jiO1SOVeRI7ry7W7mf7RGi4f0pFL0zr4jiO1TEVBRMq1N6+AW2Zl0bVVI351yQDfcSQCNHwkImVyzvGLV7LZnXeUV689jUb19HaRCNRTEJEy/e3LjXywfCd3XtCPEzs08x1HIkRFQUS+Y9nWA/z6reWc07cN15/W1XcciSAVBRH5lsMFRdyUsZDmDZJ59IqBWg47wWiQUES+5YE3lrF+Vx4vjj+VVo3r+Y4jEaaegoiE/D17K7Pmb+bnZ/dgRM/WvuOIByoKIgLA5j2HuXvuYoZ0bs7kc3v7jiOeqCiICIXFJUzMyASDJ8emkZykt4ZEpTkFEeGJ91eRtXkfT189hE4tG/qOIx7p44BIgvts9S6e+WQt44Z24sKB7X3HEc9UFEQS2K5DR7lldhY9Uxsz9SItYyEaPhJJWCUljttmZ7P/SCEvjB9Kg5Qk35EkCqinIJKgnvt8PZ+syuWXF/ajb7umvuNIlFBREElAi3P28/A7KzhvQFuuGdbFdxyJIioKIgnm0NEiJmYsJLVxPR6+XMtYyLdpTkEkwUx9bQmb9hxm5oThNG+Y4juORBmvPQUz62RmH5nZcjNbamY3+8wjEu/mLsxhbuYWJo3sxdBuLX3HkSjku6dQBNzmnFtoZk2ABWb2vnNumedcInFn/a487n1tCUO7tWTiOb18x5Eo5bWn4Jzb5pxbGLx9EFgO6CKwIjWsoKiEiRkLSalbhyfHDiapjuYRpGy+ewohZtYVSAPm+U0iEn8eeWcFS7YcYMaPTqZ9swa+40gUi4qzj8ysMTAHmOycO1DG9glmNt/M5ufm5kY+oEgM+2jlTp79bD0/Ht6FUQPa+Y4jUc57UTCzZAIF4UXn3Nyy9nHOzXDOpTvn0lNTUyMbUCSG7TyQz5TZ2fRt14S7R/fzHUdigNfhIwucIP1nYLlz7gmfWUTiTUmJ45bZWeQVFDHr6mHUT9YyFlIx3z2F04AfAeeYWVbwZ7TnTCJx4ZlP1/L5mt3c/4MB9GzTxHcciRFeewrOuc8AnQYhUsMWbtrL4++t4sKB7bnqlE6+40gM8d1TEJEadiC/kEkZmbRvVp/fXHaSlrGQKomaU1JFpPqcc9w9dzHb9ufz8o3DaVo/2XckiTHqKYjEkdnzN/Pmom3c+v3eDOncwncciUEqCiJxYs3Og9z/xjJO69mKn53Vw3cciVEqCiJxIL+wmJteyqRBShK/vXIwdbSMhYRJcwoiceA3by1nxfaDPH/dKbRpWt93HIlh6imIxLj3lm7nr19uZPzp3fhe3za+40iMU1EQiWHb9h/h9jmLOLFDU24/v4/vOBIHVBREYlRxiWPyzCwKikp4amwa9epqGQupPs0piMSo3/1zDfPW7+HxMYPontrYdxyJE+opiMSgrzfs4ckPV/HDtA5cfnJH33EkjqgoiMSYfYcLuDkjk84tG/LgpSf6jiNxRsNHIjHEOccdcxaRe+goc342gsb19CcsNUs9BZEY8n/zNvHu0h3cfl5fBnZs7juOxCEVBZEYsWL7AR58cxln9U5l/OndfMeROKWiIBIDjhQUM/GlTJrWT+axMYO0jIXUGg1IisSAX725jNU7D/HC+KGkNqnnO47EMfUURKLcW4u3kfHvTdx4Vg/O6JXqO47EORUFkSiWs/cwd85ZxKBOzbltVG/fcSQBqCiIRKmi4hJunpmFczB9bBrJSfpzldqnOQWRKDXtg9Us2LiXp8al0blVQ99xJEHoo4dIFPpi7S6e/ngNV6Z35OJBJ/iOIwlERUEkyuzJK+CWWVl0a92I+y8e4DuOJBgVBZEo4pzjFy9nszevkOnj0miYohFeiSwVBZEo8vznG/hwxU7uHt2XASc08x1HEpCKgkiUWLJlPw+9vYJz+7Xh2hFdfceRBKWiIBIF8o4WMSkjkxaNknnkikGYaRkL8UMDliJR4L43lrJ+dx4v3TCMlo1SfMeRBKaegohnr2dt4ZUFOUz8Xk+G92jlO44kOBUFEY827s7jnleXkN6lBZNG9vIdR0RFQcSXgqISJmVkUsdg2tjB1NUyFhIFNKcg4snj760kO2c/f/ivIXRsoWUsJDroo4mIB5+uyuWPn67j6lM7c8FJ7X3HEQlRURCJsNyDR7l1dja92zZm6kX9fccR+RYNH4lEUEmJ49bZWRzML+Sl/z6V+slJviOJfIv3noKZnW9mK81sjZnd6TuPSG3607/W8a/Vu5j6g/70btvEdxyR7/BaFMwsCXgauADoD4wzM/WnJS5lb97Ho++u5IIT23H10M6+44iUyXdPYSiwxjm3zjlXAMwELvGcSaTGHcwvZGJGJm2b1uehywZqGQuJWr7nFDoAm0v9ngOcWhtP9IPpn5FfWFwbhxap0MH8InYezGf2T4fTrGGy7zgi5fJdFMr6uOS+s5PZBGACQOfO4XW7e6Q2oqC4JKzHitSE8wa0I71rS98xRI7Ld1HIATqV+r0jsPXYnZxzM4AZAOnp6d8pGpUxbWxaOA8TEUkovucUvgZ6mVk3M0sBxgJveM4kIpKwvPYUnHNFZnYT8C6QBDznnFvqM5OISCLzPXyEc+4t4C3fOURExP/wkYiIRBEVBRERCVFREBGREBUFEREJUVEQEZEQcy6s74J5Y2a5wMYwH94a2FWDcWqa8lWP8lWP8lVPNOfr4pxLrcyOMVcUqsPM5jvn0n3nKI/yVY/yVY/yVU+056ssDR+JiEiIioKIiIQkWlGY4TtABZSvepSvepSveqI9X6Uk1JyCiIgcX6L1FERE5DjiriiY2RgzW2pmJWaWfsy2u8xsjZmtNLPzynl8SzN738xWB/9tUYtZZ5lZVvBng5lllbPfBjNbHNxvfm3lKeN57zezLaUyji5nv/ODbbrGzO6MYL5HzWyFmS0ys1fNrHk5+0W0/SpqDwt4Krh9kZkNqe1MpZ67k5l9ZGbLg38nN5exz9lmtr/U6z41UvmCz3/c18tz+/Up1S5ZZnbAzCYfs4/X9qs251xc/QD9gD7Ax0B6qfv7A9lAPaAbsBZIKuPxjwB3Bm/fCTwcodyPA1PL2bYBaO2hLe8HplSwT1KwLbsDKcE27h+hfKOAusHbD5f3WkWy/SrTHsBo4G0CVx4cBsyL4GvaHhgSvN0EWFVGvrOBNyP9/62yr5fP9ivjtd5O4DsAUdN+1f2Ju56Cc265c25lGZsuAWY6544659YDa4Ch5ez31+DtvwKX1k7S/7DAVdyvBDJq+7lqwVBgjXNunXOuAJhJoA1rnXPuPedcUfDXrwhcuc+3yrTHJcDfXMBXQHMzax+JcM65bc65hcHbB4HlBK6VHku8td8xRgJrnXPhfpk2KsVdUTiODsDmUr/nUPYfQ1vn3DYI/AEBbSKQ7Qxgh3NudTnbHfCemS0IXq86km4KdtGfK2corbLtWtuuJ/DpsSyRbL/KtEdUtJmZdQXSgHllbB5uZtlm9raZDYhosIpfr6hoPwJXiizvg5zP9qsW7xfZCYeZfQC0K2PTPc6518t7WBn31fqpV5XMOo7j9xJOc85tNbM2wPtmtsI592lt5wP+ADxIoJ0eJDDEdf2xhyjjsTXWrpVpPzO7BygCXiznMLXWfmWoTHt4+b/4rQBmjYE5wGTn3IFjNi8kMCRyKDiP9BrQK4LxKnq9oqH9UoCLgbvK2Oy7/aolJouCc+7cMB6WA3Qq9XtHYGsZ++0ws/bOuW3BLunOcDJ+o6KsZlYXuAw4+TjH2Br8d6eZvUpgiKJG3tQq25Zm9ifgzTI2VbZdw1KJ9rsWuAgY6YIDumUco9barwyVaY9abbOKmFkygYLwonNu7rHbSxcJ59xbZvZ7M2vtnIvIuj6VeL28tl/QBcBC59yOYzf4br/qSqThozeAsWZWz8y6Eajc/y5nv2uDt68Fyut51JRzgRXOuZyyNppZIzNr8s1tApOrS2o50zfPXXqc9oflPO/XQC8z6xb89DSWQBtGIt/5wB3Axc65w+XsE+n2q0x7vAH8OHgWzTBg/zdDlrUtOH/1Z2C5c+6JcvZpF9wPMxtK4H1id4TyVeb18tZ+pZTbu/fZfjXC90x3Tf8QePPKAY4CO4B3S227h8CZISuBC0rd/yzBM5WAVsCHwOrgvy1rOe9fgBuPue8E4K3g7e4EzmDJBpYSGDaJVFu+ACwGFhH4Q2x/bL7g76MJnMWyNsL51hAYW84K/jwTDe1XVnsAN37zOhMY/ng6uH0xpc6Si0C20wkMtSwq1W6jj8l3U7CtsglM4I+IYL4yX69oab/g8zck8CbfrNR9UdF+NfGjbzSLiEhIIg0fiYhIBVQUREQkREVBRERCVBRERCRERUFEREJUFEREJERFQUREQlQUREQkREVBRERCVBREwmRmr5mZM7OJZWx7MLjtWR/ZRMKlZS5EwmRmLYFMoC0w3DmXGbx/JPAesAI4xZWzWJ9INFJREKkGMxsBfAKsB4YQWCwtG2hGoCAs9RhPpMo0fCRSDc65L4BfEliK/Y/A/xG4KNAkFQSJReopiFRTcO38t4HzgndlOOeu9hhJJGzqKYhUkwt8snq11F3TfGURqS71FESqycx6EbgubyGBuYSlwFDnXL7XYCJhUE9BpBrMrB4wC2hE4NKbvwFOQr0FiVEqCiLV8xiQBjzinHsPuA/4HPipmV3pNZlIGDR8JBImM7uUwFzCPOB051xR8P5OBK59XBdIc86t85dSpGpUFETCYGadCbzx1yHwxr/+mO2XAK8BXxMoGAWRTylSdSoKIiISojkFEREJUVEQEZEQFQUREQlRURARkRAVBRERCVFREBGREBUFEREJUVEQEZEQFQUREQlRURARkZD/B+HOe8ka0e1dAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(range(-10, 10), [relu(i) for i in range(-10, 10)])\n",
    "plt.xlabel('x', fontsize = 20)\n",
    "plt.ylabel('relu', fontsize = 20);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-06-21T03:15:44.046735Z",
     "start_time": "2019-06-21T03:15:43.888331Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAFOVJREFUeJzt3X+s3fV93/HnC9v8UJcUMi6tsZ0ZtSYasMRJTq14a1bXsOCxCveH0rnqNCYqsUROlkRqolhM0ShC5OfSTWsWsTYbXZMQd6EbQWIJsN5ESMbOdWpTTHBxBi0ODN9oop3/mCk37/1xvp4Pl2ufc82991x/8nxIR+f7/Xw/3+95n8vldT738/0ef1NVSJLadd64C5AkLS6DXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktS4leMuAODSSy+t9evXj7sMSTqn7N+//wdVNTGs37II+vXr1zM1NTXuMiTpnJLkz0fp59SNJDXOoJekxhn0ktQ4g16SGjdy0CdZkeRPktzfrb8hyYNJnuqeLxnouyvJkSSHk1y/GIVLkkYznxH9B4DvDqx/FHi4qjYAD3frJLkK2AFcDWwDPpdkxcKUK0mar5GCPsla4B8BvzvQvB24u1u+G/jFgfZ7qupEVT0NHAE2LUy5kv6/PXvgzjv7z9IZjHod/W8DHwFeN9D2E1X1PEBVPZ/ksq59DfDoQL+jXZukhbJnD1x7Lbz0Epx/Pjz8MGzePO6qtEwNHdEn+QXgWFXtH/GYmaPtVTemTXJLkqkkU9PT0yMeWhIAk5P9kJ+Z6T9PTo67Ii1jo0zd/D3gxiTPAPcAW5P8AfBCktUA3fOxrv9RYN3A/muB52YftKruqqpeVfUmJoZ+g1fSoC1b+iP5FSv6z1u2jLsiLWNDg76qdlXV2qpaT/8k6/+oqn8C3Afc1HW7Cfhv3fJ9wI4kFyS5AtgA7FvwyqUfZZs396drbr/daRsN9Vr+rZuPA7uT/AbwF8C7AarqUJLdwBPAy8DOqpp5zZVKeqXNmw14jSRVr5o+X3K9Xq/8R80kaX6S7K+q3rB+fjNWkhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNGxr0SS5Msi/JwSSHktzWtb8lyZ4kf5rka0leP7DPriRHkhxOcv1ivgFJ0pmNMqI/AWytqrcAG4FtSd4B/C7w0ar6O8AfAR8GSHIVsAO4GtgGfC7JisUoXpI03NCgr77j3eqq7lHAm4Bvde0PAr/SLW8H7qmqE1X1NHAE2LSgVUuSRjbSHH2SFUkOAMeAB6tqL/A4cGPX5d3Aum55DfDswO5Hu7bZx7wlyVSSqenp6bOtX5I0xEhBX1UzVbURWAtsSnINcDOwM8l+4HXAS133zHWIOY55V1X1qqo3MTFxdtVLkoaa11U3VfUiMAlsq6onq+pdVfV24MvA97puRzk1uof+h8NzC1CrJOksjHLVzUSSi7vli4DrgCeTXNa1nQf8S+Dz3S73ATuSXJDkCmADsG8xipckDbdyhD6rgbu7K2fOA3ZX1f1JPpBkZ9fnXuA/AlTVoSS7gSeAl4GdVTWzCLVLkkaQqldNny+5Xq9XU1NT4y5Dks4pSfZXVW9YP78ZK0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDVuaNAnuTDJviQHkxxKclvXvjHJo0kOJJlKsmlgn11JjiQ5nOT6xXwDkqQzWzlCnxPA1qo6nmQV8EiSB4DfAm6rqgeS3AB8EtiS5CpgB3A1cDnwUJIrq2pmkd6DJOkMho7oq+94t7qqe1T3eH3X/uPAc93yduCeqjpRVU8DR4BNSJLGYpQRPUlWAPuBnwZ+p6r2Jvkg8PUkn6b/gfF3u+5rgEcHdj/atc0+5i3ALQBvfOMbz/oNSJLObKSTsVU1U1UbgbXApiTXAO8FPlRV64APAb/Xdc9ch5jjmHdVVa+qehMTE2dXvSRpqHlddVNVLwKTwDbgJuDebtMfcmp65iiwbmC3tZya1pEkLbFRrrqZSHJxt3wRcB3wJP3w/rmu21bgqW75PmBHkguSXAFsAPYtdOGSpNGMMke/Gri7m6c/D9hdVfcneRH4N0lWAv+Xbr69qg4l2Q08AbwM7PSKG0kan1S9avp8yfV6vZqamhp3GZJ0Tkmyv6p6w/r5zVhJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcUODPsmFSfYlOZjkUJLbuvavJDnQPZ5JcmBgn11JjiQ5nOT6xXwDkqQzWzlCnxPA1qo6nmQV8EiSB6rqH5/skOQzwF92y1cBO4CrgcuBh5JcWVUzC1++JGmYoSP66jvera7qHnVye5IAvwp8uWvaDtxTVSeq6mngCLBpQauWJI1spDn6JCu6qZljwINVtXdg8zuBF6rqqW59DfDswPajXZskaQxGCvqqmqmqjcBaYFOSawY2/xqnRvMAmesQsxuS3JJkKsnU9PT0fGqWJM3DvK66qaoXgUlgG0CSlcAvA18Z6HYUWDewvhZ4bo5j3VVVvarqTUxMzLNsSWrAnj1w553950U09GRskgngr6vqxSQXAdcBn+g2Xwc8WVVHB3a5D/hSkn9N/2TsBmDfwpYtSee4PXvg2mvhpZfg/PPh4Ydh8+ZFealRRvSrgT9O8hjwbfpz9Pd323bwymkbquoQsBt4AvjvwE6vuJGkWSYn+yE/M9N/npxctJcaOqKvqseAt55m2z87TfsdwB2vqTJJatmWLf2R/MkR/ZYti/ZSo1xHL0laaJs396drJif7Ib9I0zZg0EvS+GzevKgBf5L/1o0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1bmjQJ7kwyb4kB5McSnLbwLb3JznctX9yoH1XkiPdtusXq3hJ0nCj3ErwBLC1qo4nWQU8kuQB4CJgO/DmqjqR5DKAJFcBO4CrgcuBh5JcWVUzi/MWJElnMnREX33Hu9VV3aOA9wIfr6oTXb9jXZ/twD1VdaKqngaOAJsWvHJJ0khGmqNPsiLJAeAY8GBV7QWuBN6ZZG+Sbyb5ma77GuDZgd2Pdm2SpDEYKeiraqaqNgJrgU1JrqE/7XMJ8A7gw8DuJAEy1yFmNyS5JclUkqnp6emzfgOSpDOb11U3VfUiMAlsoz9Sv7eb2tkH/BC4tGtfN7DbWuC5OY51V1X1qqo3MTFxluVLkoYZ5aqbiSQXd8sXAdcBTwL/FdjatV8JnA/8ALgP2JHkgiRXABuAfYtTviRpmFGuulkN3J1kBf0Pht1VdX+S84EvJHkceAm4qaoKOJRkN/AE8DKw0ytuJGl80s/m8er1ejU1NTXuMiTpnJJkf1X1hvXzm7GS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS44YGfZILk+xLcjDJoSS3de3/Ksn3kxzoHjcM7LMryZEkh5Ncv5hvQJJ0ZitH6HMC2FpVx5OsAh5J8kC37bNV9enBzkmuAnYAVwOXAw8lubKqZhaycEnSaIaO6KvveLe6qnvUGXbZDtxTVSeq6mngCLDpNVcqSTorI83RJ1mR5ABwDHiwqvZ2m96X5LEkX0hySde2Bnh2YPejXZskaQxGCvqqmqmqjcBaYFOSa4B/D/wUsBF4HvhM1z1zHWJ2Q5JbkkwlmZqenj6r4iVJw83rqpuqehGYBLZV1QvdB8APgf/AqemZo8C6gd3WAs/Ncay7qqpXVb2JiYmzKl6SNNwoV91MJLm4W74IuA54MsnqgW6/BDzeLd8H7EhyQZIrgA3AvoUtW5I0qlGuulkN3J1kBf0Pht1VdX+S/5xkI/1pmWeAfw5QVYeS7AaeAF4GdnrFjSSNT6rOdAHN0uj1ejU1NTXuMiTpnJJkf1X1hvXzm7GS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWrc0KBPcmGSfUkOJjmU5LZZ238zSSW5dKBtV5IjSQ4nuX4xCpckjWblCH1OAFur6niSVcAjSR6oqkeTrAP+AfAXJzsnuQrYAVwNXA48lOTKqppZhPolSUMMHdFX3/FudVX3qG79s8BHBtYBtgP3VNWJqnoaOAJsWriSJUnzMdIcfZIVSQ4Ax4AHq2pvkhuB71fVwVnd1wDPDqwf7dpmH/OWJFNJpqanp8+yfEnSMCMFfVXNVNVGYC2wKcmbgVuBj83RPXMdYo5j3lVVvarqTUxMzKdmSdI8zOuqm6p6EZikPz1zBXAwyTP0PwC+k+Qn6Y/g1w3sthZ4biGKlSTN3yhX3Uwkubhbvgi4DviTqrqsqtZX1Xr64f62qvpfwH3AjiQXJLkC2ADsW7R3IEk6o1GuulkN3J1kBf0Pht1Vdf/pOlfVoSS7gSeAl4GdXnEjSeMzNOir6jHgrUP6rJ+1fgdwx2uqTJK0IPxmrCQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktS4oUGf5MIk+5IcTHIoyW1d++1JHktyIMk3klw+sM+uJEeSHE5y/WK+AUnSmY0yoj8BbK2qtwAbgW1J3gF8qqreXFUbgfuBjwEkuQrYAVwNbAM+l2TFolQvSRpqaNBX3/FudVX3qKr6q4FuPwZUt7wduKeqTlTV08ARYNMC1vxKe/bAnXf2nyVJr7JylE7diHw/8NPA71TV3q79DuCfAn8J/HzXfQ3w6MDuR7u2hbdnD1x7Lbz0Epx/Pjz8MGzevCgvJUnnqpFOxlbVTDdFsxbYlOSarv3WqloHfBF4X9c9cx1idkOSW5JMJZmanp4+u+onJ/shPzPTf56cPLvjSFLD5nXVTVW9CEzSn3sf9CXgV7rlo8C6gW1rgefmONZdVdWrqt7ExMR8yjhly5b+SH7Fiv7zli1ndxxJatgoV91MJLm4W74IuA54MsmGgW43Ak92y/cBO5JckOQKYAOwb2HL7mze3J+uuf12p20k6TRGmaNfDdzdzdOfB+yuqvuTfDXJm4AfAn8OvAegqg4l2Q08AbwM7KyqmcUpn364G/CSdFqpetX0+ZLr9Xo1NTU17jIk6ZySZH9V9Yb185uxktQ4g16SGmfQS1LjDHpJapxBL0mNWxZX3SSZpn+J5tm6FPjBApWzkKxrfqxrfqxrflqs629V1dBvnC6LoH+tkkyNconRUrOu+bGu+bGu+flRrsupG0lqnEEvSY1rJejvGncBp2Fd82Nd82Nd8/MjW1cTc/SSpNNrZUQvSTqNZRn0Sb6Q5FiSx2e1v7+74fihJJ8caJ/zZuRJ3p7kT7tt/zbJXDdFeU11JflKd4P0A0meSXJgmdS1McmjXV1TSTYNbBtnXW9Jsqd7na8lef0Y6lqX5I+TfLf7XfpA1/6GJA8meap7vmQpaztDXe/u1n+YpDdrn3HW9akkTyZ5LMkfpfvnzJdBXbd3NR1I8o0kly+Huga2/2aSSnLpktVVVcvuAfx94G3A4wNtPw88BFzQrV/WPV8FHAQuAK4Avges6LbtAzbTv+vVA8A/XOi6Zm3/DPCx5VAX8I2TxwVuACaXSV3fBn6uW74ZuH0Mda0G3tYtvw74s+71Pwl8tGv/KPCJpaztDHX9beBN9G/60xvoP+663gWs7No/sYx+Xq8f6PMvgM8vh7q69XXA1+l/b+jSpaprWY7oq+pbwP+e1fxe4ONVdaLrc6xrn/Nm5ElW0/8Pvqf6P7HfB35xEeoCoPuk/VXgy8ukrgJOjpZ/nFN3+Rp3XW8CvtUtP8ipO5MtZV3PV9V3uuX/A3yX/n2NtwN3d93uHnidJantdHVV1Xer6vAcu4y7rm9U1ctdt0fp301uOdT1VwPdfoxTtzIda13d5s8CH+GVt1dd9LqWZdCfxpXAO5PsTfLNJD/Tta8Bnh3od/Jm5Gu65dnti+WdwAtV9dQyqeuDwKeSPAt8Gti1TOp6nP4dyQDezanbTo6lriTrgbcCe4GfqKrnof8/K3DZuGqbVdfpLKe6bqY/4lwWdSW5o/vd/3XgY8uhriQ3At+vqoOzui16XedS0K8ELgHeAXwY2N2Nok93M/KRblK+gH6NU6N5zvD6S1XXe4EPVf/m7R8Cfm+Z1HUzsDPJfvp/1r40rrqS/A3gq8AHZ40CX9V1KWs71+pKciv9u8l9cbnUVVW3dr/7XwTeN+666P98buXUh84rui52XedS0B8F7q2+ffRvYXgpp78Z+VFO/Sk52L7gkqwEfhn4yqx6x1nXTcC93fIfAidPxo61rqp6sqreVVVvp//B+L1x1JVkFf3/Cb9YVSd/Ti90fy7TPZ+cHlyy2k5T1+mMva4kNwG/APx6N72wLOoa8CVOTQ+Os66foj//fjDJM91rfCfJTy5JXWd7wmGxH8B6XnkS7z3Ab3XLV9L/UyfA1bzyRMb/5NSJjG/T/wvg5ImMGxa6rq5tG/DNWW1jrYv+vOCWbvlaYP8yqevkSfTz6M853rzUdXXH+X3gt2e1f4pXnoz95FLWdrq6BrZP8sqTsWOtq/u9fwKYGMfv/hnq2jCw/H7gvyyHumb1eYZTJ2MXva7X9D/xYj3oj/SeB/6a/qfabwDnA39Af473O8DWgf630h8ZHmbgrDTQ6/p/D/h3dF8QW8i6uvb/BLxnjv5jqwv4WWB/9wu0F3j7MqnrA/SvQvgz4OODr7GEdf0s/T+BHwMOdI8bgL8JPAw81T2/YSlrO0Ndv9T9/E4ALwBfXyZ1HaE/4DrZ9vllUtdXu9d4DPga/RO0Y69rVp9n6IJ+Kerym7GS1LhzaY5eknQWDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhr3/wDxQoGQo4t5dgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "x_train = np.array([[2104],[1600],[2400]], dtype=np.float32)\n",
    "y_train = np.array([[399.900], [329.900], [369.000]], dtype=np.float32)\n",
    "\n",
    "plt.plot(x_train, y_train, 'r.')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-06-20T12:55:43.595266Z",
     "start_time": "2019-06-20T12:55:43.591528Z"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "# x_train = np.array([[3.3], [4.4], [5.5], [6.71], [6.93], [4.168],\n",
    "#                     [9.779], [6.182], [7.59], [2.167], [7.042],\n",
    "#                     [10.791], [5.313], [7.997], [3.1]], dtype=np.float32)\n",
    "\n",
    "# y_train = np.array([[1.7], [2.76], [2.09], [3.19], [1.694], [1.573],\n",
    "#                     [3.366], [2.596], [2.53], [1.221], [2.827],\n",
    "#                     [3.465], [1.65], [2.904], [1.3]], dtype=np.float32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-06-21T03:15:47.861042Z",
     "start_time": "2019-06-21T03:15:47.858014Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "x_train = torch.from_numpy(x_train)\n",
    "y_train = torch.from_numpy(y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-07T04:47:17.479477Z",
     "start_time": "2019-04-07T04:47:17.468259Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**nn.Linear**\n",
    "\n",
    "> help(nn.Linear)\n",
    "\n",
    "Applies a linear transformation to the incoming data: $y = xA^T + b$\n",
    "\n",
    "- **in_features**: size of each input sample\n",
    "- **out_features**: size of each output sample\n",
    "- **bias**: If set to False, the layer will not learn an additive bias. Default: ``True``"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-06-21T03:20:09.742024Z",
     "start_time": "2019-06-21T03:20:09.731709Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Linear Regression Model\n",
    "class LinearRegression(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(LinearRegression, self).__init__()\n",
    "        self.linear = nn.Linear(1, 1)  # input and output is 1 dimension\n",
    "\n",
    "    def forward(self, x):\n",
    "        out = self.linear(x)\n",
    "        return out\n",
    "\n",
    "model = LinearRegression()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-06-20T12:56:00.581797Z",
     "start_time": "2019-06-20T12:56:00.577795Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Define Loss and Optimizatioin function\n",
    "criterion = nn.MSELoss()\n",
    "optimizer = optim.SGD(model.parameters(), lr=1e-9)#1e-4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-06-20T13:07:42.012941Z",
     "start_time": "2019-06-20T13:07:42.003484Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "> help(nn.MSELoss)\n",
    "\n",
    "To measures the **mean squared error** (squared L2 norm) between each element in the input `x` and target `y`.\n",
    "\n",
    "> help(optim.SGD)\n",
    "\n",
    "Implements **stochastic gradient descent** (optionally with momentum)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-06-21T03:26:16.906840Z",
     "start_time": "2019-06-21T03:26:16.759326Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch[50/1000], loss: 625150.562500\n",
      "Epoch[100/1000], loss: 625150.562500\n",
      "Epoch[150/1000], loss: 625150.562500\n",
      "Epoch[200/1000], loss: 625150.562500\n",
      "Epoch[250/1000], loss: 625150.562500\n",
      "Epoch[300/1000], loss: 625150.562500\n",
      "Epoch[350/1000], loss: 625150.562500\n",
      "Epoch[400/1000], loss: 625150.562500\n",
      "Epoch[450/1000], loss: 625150.562500\n",
      "Epoch[500/1000], loss: 625150.562500\n",
      "Epoch[550/1000], loss: 625150.562500\n",
      "Epoch[600/1000], loss: 625150.562500\n",
      "Epoch[650/1000], loss: 625150.562500\n",
      "Epoch[700/1000], loss: 625150.562500\n",
      "Epoch[750/1000], loss: 625150.562500\n",
      "Epoch[800/1000], loss: 625150.562500\n",
      "Epoch[850/1000], loss: 625150.562500\n",
      "Epoch[900/1000], loss: 625150.562500\n",
      "Epoch[950/1000], loss: 625150.562500\n",
      "Epoch[1000/1000], loss: 625150.562500\n"
     ]
    }
   ],
   "source": [
    "num_epochs = 1000\n",
    "for epoch in range(num_epochs):\n",
    "    inputs = Variable(x_train)\n",
    "    target = Variable(y_train)\n",
    "\n",
    "    # forward\n",
    "    out = model(inputs)\n",
    "    loss = criterion(out, target)\n",
    "    # backward\n",
    "    optimizer.zero_grad() # Clears the gradients of all optimized\n",
    "    loss.backward()\n",
    "    optimizer.step() # Performs a single optimization step.\n",
    "\n",
    "    if (epoch+1) % 50 == 0:\n",
    "        print('Epoch[{}/{}], loss: {:.6f}'\n",
    "              .format(epoch+1, num_epochs, loss.data.item()))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "$$\n",
    "\\ell(x, y) = L = \\{l_1,\\dots,l_N\\}^\\top, \\quad\n",
    "        l_n = \\left( x_n - y_n \\right)^2,\n",
    "$$\n",
    "    where :`N` is the batch size. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-06-20T13:31:47.227886Z",
     "start_time": "2019-06-20T13:31:47.224533Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on method eval in module torch.nn.modules.module:\n",
      "\n",
      "eval() method of __main__.LinearRegression instance\n",
      "    Sets the module in evaluation mode.\n",
      "    \n",
      "    This has any effect only on certain modules. See documentations of\n",
      "    particular modules for details of their behaviors in training/evaluation\n",
      "    mode, if they are affected, e.g. :class:`Dropout`, :class:`BatchNorm`,\n",
      "    etc.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "help(model.eval)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-07T05:26:38.112863Z",
     "start_time": "2019-04-07T05:26:38.108523Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LinearRegression(\n",
       "  (linear): Linear(in_features=1, out_features=1, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 109,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.eval()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-07T05:54:13.026497Z",
     "start_time": "2019-04-07T05:54:12.777585Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZEAAAETCAYAAAD3WTuEAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJzt3Xl8VNX5x/HPY1gCAoKIgGyhij9ZjRpXfloFF2oRrNL+sLjghlWpuGFVrFTRFoWKtIKKQgWbVlREqKKCAu6y7yASNEBQISCgNIIs5/fHuUMWCJAhM3cy832/XvPKPWfuzDyZLM8899x7jjnnEBERicZhYQcgIiIVl5KIiIhETUlERESipiQiIiJRUxIREZGoKYmIiEjUlERERCRqSiIiIhI1JREREYlapbADiLWjjjrKZWRkhB2GiEiFMmfOnA3OuXoH2i/pk0hGRgazZ88OOwwRkQrFzFYdzH46nCUiIlFTEhERkagpiYiISNSSfkxkX3bs2EFeXh7btm0LOxQB0tPTady4MZUrVw47FBEpo5RMInl5edSsWZOMjAzMLOxwUppzjo0bN5KXl0fz5s3DDkdEyiglD2dt27aNunXrKoEkADOjbt26qgpFykmDBmC2961Bg9i8XkomEUAJJIHoZyFSftatK1v/oUrZJCIiIodOSSQkeXl5dO3alRYtWnDsscfSp08ffvrpp33u+/XXX9OtW7cDPufFF1/M5s2bo4rnT3/6E4MHDz7gfjVq1Njv/Zs3b2b48OFRxSAiFY+SyMHIzoaMDDjsMP81O/uQns45x2WXXcall17KihUr+OKLL9i6dSv9+vXba9+dO3dyzDHH8Oqrrx7weSdNmkTt2rUPKbZDpSQiEo4tW6BXr/i/rpLIgWRn+5/MqlXgnP/aq9chJZKpU6eSnp7OtddeC0BaWhpDhgxh1KhRFBQU8MILL9ClSxc6dOhAx44dyc3NpU2bNgAUFBTwm9/8hlatWvGrX/2K008/fc+0LhkZGWzYsIHc3FxatmzJjTfeSOvWrbnwwgv58ccfAXjuuec49dRTOfHEE7n88sspKCjYb6xfffUVZ555Jm3btuWBBx7Y079161Y6duzIySefTNu2bZkwYQIA9957LytXriQzM5O+ffuWup+IlJ+JE6FVKxg5Mv6vnRBJxMzSzGyemb0RtLPNbLmZLTazUWZWOeg3M/ubmeWY2UIzOznmwfXrByX/0RYU+P4oLVmyhFNOOaVYX61atWjatCk5OTkAzJ07l1dffZX333+/2H7Dhw+nTp06LF26lAEDBjBnzpx9vsaKFSu49dZbWbJkCbVr12bcuHEAXHbZZcyaNYsFCxbQsmVLRh7gt65Pnz7cfPPNLFq0iIYNG+7pT09PZ/z48cydO5dp06Zx11134Zxj4MCBHHvsscyfP59BgwaVup+IHLr166F7d+jaFerWhRkzoH79fe9bWv+hSogkAvQBlhVpZwMnAG2BasANQf8vgBbBrRfwdMwjW726bP3l5IILLuDII4/cq/+jjz6ie/fuALRp04Z27drt8/HNmzcnMzMTgFNOOYXc3FwAFi9ezNlnn03btm3Jzs5myZIl+43j448/5oorrgDgqquu2tPvnOP++++nXbt2nH/++axdu5Z1+zj942D3E5GD55w/GNKqFYwfDwMGwOzZkJUF337r7y95+/bb2MQSehIxs8bAL4HnI33OuUkuAMwEGgd3dQXGBHd9BtQ2s4Z7PWl5atq0bP0HoVWrVntVEN9//z2rV6/muOOOA+Dwww+P+vkBqlatumc7LS2NnTt3AtCzZ0+eeuopFi1aRP/+/Q/q+ox9nYKbnZ1Nfn4+c+bMYf78+dSvX3+fz3Ww+4nIwVmzBi65BK68Elq0gHnz4IEHoEqVcOIJPYkATwL3ALtL3hEcxroKeDvoagSsKbJLXtBX8nG9zGy2mc3Oz88/tOgefRSqVy/eV726749Sx44dKSgoYMyYMQDs2rWLu+66i549e1K95GuV0L59e15++WUAli5dyqJFi8r02j/88AMNGzZkx44dZB/EuE779u156aWXAIrtv2XLFo4++mgqV67MtGnTWLXKzxpds2ZNfvjhhwPuJyJls3s3PPMMtG4N06bBk0/CRx/5aiRMoSYRM+sMrHfO7fvAPgwHPnDOfViW53XOjXDOZTnnsurVO+CaKvvXoweMGAHNmvnLPps18+0ePaJ+SjNj/PjxvPLKK7Ro0YLjjz+e9PR0/vznPx/wsbfccgv5+fm0atWKBx54gNatW3PEEUcc9GsPGDCA008/nfbt23PCCScccP+hQ4cybNgw2rZty9q1a/f09+jRg9mzZ9O2bVvGjBmz57nq1q1L+/btadOmDX379i11PxE5eCtWQIcOcPPNcPrpsHgx9OkDaWlhRwYW5iCnmf0FX2nsBNKBWsBrzrkrzaw/cBJwmXNud7D/s8B059y/g/Zy4Fzn3DelvUZWVpYruSjVsmXLaNmyZSy+pZjbtWsXO3bsID09nZUrV3L++eezfPlyqoRVy5aTivwzEYmVnTthyBB48EGoWhWeeAKuvdZ/no01M5vjnMs60H6hTsDonLsPuA/AzM4F7g4SyA3ARUDHSAIJTAR6m9lLwOnAlv0lkGRUUFDAeeedx44dO3DOMXz48AqfQERkbwsWwPXXw5w5cOmlMGwYHHNM2FHtLVFn8X0GWAV8GgzqvuacexiYBFwM5AAFwLWhRRiSmjVrarlfkSS2fTs88ggMHAhHHgkvvwzdusWn+ohGwiQR59x0YHqwvc+4grO1bo1fVCIi8fPpp776WLYMrr7aH76qWzfsqPYvEc7OEhFJaf/9L9x+O7RvD1u3wqRJMHp04icQSKBKREQkFb37Ltx4I+Tmwq23wl/+AjVrhh3VwVMlIiISgk2b/KGrCy7wFwp+8AE89VTFSiCgJBKatLQ0MjMz99xyc3OZPXs2t912GwDTp0/nk08+2bP/66+/ztKlS/e0H3zwQd59991yiSUycWNREydOZODAgeXy/CJS3Pjx/iLB0aPh3nv9mVhnnx12VNHR4awDaNBg3yuC1a9/aHPRVKtWjfnz5xfry8jIICvLn5Y9ffp0atSowVlnnQX4JNK5c2daBZenPvzww9G/+EHo0qULXbp0ielriKSadevg97+HV16BzEx48004OfbTyMaUKpEDiOdSk9OnT6dz587k5ubyzDPPMGTIEDIzM3n//feZOHEiffv2JTMzk5UrV9KzZ889a4xkZGTQv3//PdOtf/755wDk5+dzwQUX0Lp1a2644QaaNWu2V8VRmhdeeIHevXsDfr6t2267jbPOOouf/exnxdY2GTRoEKeeeirt2rWjf//+5fyOiCQH52DMGGjZEiZM8LMmzZxZ8RMIqBLh9tuhREFw0M49d9/9mZl+Xpv9+fHHH/fMstu8eXPGjx+/576MjAx+97vfUaNGDe6++27AVwadO3cudYXDo446irlz5zJ8+HAGDx7M888/z0MPPUSHDh247777ePvttw847fv+fPPNN3z00Ud8/vnndOnShW7dujF58mRWrFjBzJkzcc7RpUsXPvjgA84555yoX0ck2axeDTfdBG+/DWed5df8SKbZf1I+iYRlX4ezDsVll10G+GnfX3vtNcBPGx9JTp06daJOnTpRP/+ll17KYYcdRqtWrfZM5T558mQmT57MSSedBPiFqlasWKEkIoKfMPHpp/2Yh3Pw97/DLbf4BVKTSconkQNVDPu7SnT69HIN5ZBEpn4vOu17LJ4f2LOolHOO++67j5tuuqncX0+kIlu+HG64wc+ye+GF8OyzfmXtZJRkOTF5lJxSvWT7YBSdNn7y5Mls2rSpXGO86KKLGDVqFFu3bgVg7dq1rF+/vlxfQ6Qi2bHDT1dy4omwZAm88II/jJWsCQSURA4o3ktNRlxyySWMHz+ezMxMPvzwQ7p3786gQYM46aSTWLly5UE9R//+/Zk8eTJt2rThlVdeoUGDBtQs5ST0du3a0bhxYxo3bsydd955UM9/4YUX8tvf/nbPGuzdunUrc6ITSRbz5vlp2u+7Dzp3hqVL4ZprEnfOq/IS6lTw8ZBsU8GXxfbt20lLS6NSpUp8+umn3HzzzeU6DlOeUuVnIsln2za/PO1jj8FRR/nZdi+/POyoDl2FmApeYmv16tX85je/Yffu3VSpUoXnnnsu7JBEksrHH/urzpcv9+t8DB7sZ95NJUoiSaxFixbMmzcv7DBEks7WrXD//X6akqZN4Z13/AB6KkrZMZFkP4xXkehnIRXJO+/4dc6fespffb54ceomEEjRJJKens7GjRv1zysBOOfYuHEj6enpYYcisl/ffQc9e0KnTlC9Onz4IQwdCjVqhB1ZuFLycFbjxo3Jy8sjPz8/7FAEn9QbN24cdhgipRo3zk/TvmED9OsHDzwA+tzjpWQSqVy5Ms2bNw87DBFJcN98A717w2uvwUkn+Ws+gtmKJJCSh7NERPbHOX+hYKtWfqbdgQP9hIlKIHtLiCRiZmlmNs/M3gjazc1shpnlmNlYM6sS9FcN2jnB/Rlhxi0iySc3Fy66yJ+y27YtLFwIf/gDVErJ4zYHlhBJBOgDLCvSfgwY4pw7DtgEXB/0Xw9sCvqHBPuJSHnKzvbzdBx2mP+anR12RHGxaxf87W/Qpg18+qm/aHD6dDj++LAjS2yhJxEzawz8Eng+aBvQAYgsWjEauDTY7hq0Ce7vGOwvIuUhOxt69YJVq/wxnVWrfDvJE8myZXDOOdCnj19hcMmS5JxxNxYS4S16ErgH2B206wKbnXORqWjzgEbBdiNgDUBw/5Zg/2LMrJeZzTaz2ToDS6QM+vWDgoLifQUFvj8J7djhF4jKzITPP/cLR02a5C8glIMTahIxs87AeufcnPJ8XufcCOdclnMuq169euX51CLJbfXqsvVXYHPnwqmn+tN1L73UT5h41VXJP2FieQu7EmkPdDGzXOAl/GGsoUBtM4sMYzUG1gbba4EmAMH9RwAb4xmwSFIr7SN4En00//FHv1DUaafB+vUwfjyMHRv7mbmTVahJxDl3n3OusXMuA+gOTHXO9QCmAZF1YK8BJgTbE4M2wf1TnS47Fyk/jz7qL8cuqnp1358EPvjAr/Xx2GP+6vOlS30VItELuxIpzR+AO80sBz/mEVkcfCRQN+i/E7g3pPhEklOPHjBiBDRr5o/rNGvm2z16hB3ZIfn+e3/F+c9/Djt3wrvvwvPPQ+3aYUdW8aXkeiIikjreegtuugny8vzZV488AocfHnZUiU/riYhIStu4Ee64A1580V95/skncMYZYUeVfBL1cJaISFScg5dfhpYt4d//hj/+0Z+JpQQSG6pERCRpfP21v0hwwgTIyvJjH+3ahR1VclMlIiIVnnMwcqQ/bPXOOzBokJ+6RAkk9lSJiEiF9uWXcOONMHWqP/vq+efhuOPCjip1qBIRkQpp1y548kk/0+6sWfDMMz6RKIHElyoREalwliyB66+HGTPgl7/0CUSLY4ZDlYiIVBg//QQDBvhVBnNy/OTC//mPEkiYVImISIUwa5avPhYtgiuugKFDQfOrhk+ViIgktIIC6NvXX+fx3XcwcSL8619KIIlClYiIJKzp0/2ZVzk5fm2sxx+HI44IOyopSpWIiCScLVvgd7+D887z14BMnQrPPqsEkoiUREQkobz5JrRuDc89B3fdBQsX+mQiiUlJREQSQn6+n3G+c2eoU8dfcT548N7Lm0hiURIRkVA55ydKbNUKXnkF/vQnmDPHrzwoiU8D6yISmrw8uPlmeOMNnzRGjoQ2bcKOSspClYiIxN3u3X7BxNat4b334Ikn/HofSiAVjyoREYmrnBx/2u706X7A/Lnn4Nhjw45KoqVKRETiYtcu+Otf/fTsc+f65PHee0ogFV2oScTM0s1sppktMLMlZvZQ0N/RzOaa2Xwz+8jMjgv6q5rZWDPLMbMZZpYRZvwicnAWLYIzz4S774bzz4elS+GGG8As7MjkUIVdiWwHOjjnTgQygU5mdgbwNNDDOZcJ/At4INj/emCTc+44YAjwWAgxi8hB2r4d+veHk0+G3Fx46SW/6mCjRmFHJuUl1CTivK1Bs3Jwc8GtVtB/BPB1sN0VGB1svwp0NNNnGZFENGMGnHIKPPwwdO/uq4//+z9VH8km7EoEM0szs/nAemCKc24GcAMwyczygKuAgcHujYA1AM65ncAWoO4+nrOXmc02s9n5+fnx+DZEJPDf/8Kdd/rDV1u2+NN3X3wRjjoq7MgkFkJPIs65XcFhq8bAaWbWBrgDuNg51xj4B/BEGZ9zhHMuyzmXVU9TfYrEzdSpfuB8yBA/99WSJX7RKEleoSeRCOfcZmAa8AvgxKAiARgLnBVsrwWaAJhZJfyhro1xDlVESti82Z+227EjHHaYP313+HCoVeuAD5UKLuyzs+qZWe1guxpwAbAMOMLMjg92i/QBTASuCba7AVOdcy6OIYtICRMn+osGR42Ce+7xEyb+/OdhRyXxEvbFhg2B0WaWhk9oLzvn3jCzG4FxZrYb2ARcF+w/EnjRzHKA74DuYQQtIrB+Pdx2G4wd6w9hTZgAWVlhRyXxFmoScc4tBE7aR/94YPw++rcBv45DaCJSCuf82uZ9+sDWrX7N8z/8ASpXDjsyCUPYlYiIVCBr1vgB80mT/HK1I0f62XcldSXMwLqIJK7du+Hpp/3Yx/Tp8OST8NFHSiCiSkREimjQANat27u/cmXYscNPWTJiBDRvHv/YJDEpiYjIHvtKIOATyKhR0LOnrjiX4pREROSgXHtt2BFIItKYiIiIRE1JREQAv9aHSFkpiYikuHXr/DjH3XeHHYmUm+xsyMjwc9BkZPh2jCiJiKSwe+7xZ2RFlDZfaf368YlHykF2NvTqBatW+StDV63y7RglEiURkRT05Ze++hg0yLf/8hf//2b9ev+15O3bb8ONV8qgXz8oKCjeV1Dg+2NAZ2eJpJirroJ//rOwvWkT1K4dXjxSzlavLlv/IVIlIpIiFizw1UckgYwc6asMJZAk07Rp2foPkZKISJJzzq/zkZnp27Vq+aMb1123/8dJBfXoo1C9evG+6tV9fwwoiYgksQ8+8CfoTJ3q26+/7pesrVYt3Lgkhnr08HPTNGvmS89mzXy7R4+YvJzGRESS0M6d0KYNLF/u2yecAIsWQSX9xaeGHj1iljRKUiUikmQmTPATJkYSyPvvw7JlSiASG/q1EkkSP/4IRx/tF4oC6NAB3n1XEyZKbKkSEUkCo0b5sdNIApk/H957TwlEYk+ViEgFtnkz1KlT2L7ySnjxxfDikdQTaiViZulmNtPMFpjZEjN7KOg3M3vUzL4ws2VmdluR/r+ZWY6ZLTSzk8OMXyRMjz1WPIGsXKkEIvEXdiWyHejgnNtqZpWBj8zsLaAl0AQ4wTm328yODvb/BdAiuJ0OPB18FUkZ33wDxxxT2O7bFx5/PLx4JLWFmkSccw4IjuJSObg54Gbgt8653cF+64N9ugJjgsd9Zma1zayhc+6bOIcuEoo77vDrm0d8+60mR5RwhT6wbmZpZjYfWA9Mcc7NAI4F/s/MZpvZW2bWIti9EbCmyMPzgr6Sz9kreOzs/Pz8WH8LIjGXk+MHySMJZPBgfyW6EoiELfQk4pzb5ZzLBBoDp5lZG6AqsM05lwU8B4wq43OOcM5lOeey6pU2t7VIBeAcdO8OLVoU9m3ZAnfdFV5MIkWFnkQinHObgWlAJ3yF8Vpw13igXbC9Fj9WEtE46BNJOvPm+SlLxo717dGjfVKpVSvcuESKCvvsrHpmVjvYrgZcAHwOvA6cF+z2c+CLYHsicHVwltYZwBaNh0iy2b0bzj4bTg7OPaxb119IePXV4cYlsi9lGlg3s/rOuXXl+PoNgdFmloZPaC87594ws4+AbDO7Az/wfkOw/yTgYiAHKACuLcdYREI3bZq/0jzijTfgl78MLx6RAynr2Vmrzex14Fnn3NRDfXHn3ELgpH30bwb2+tMJzsq69VBfVyTR7NjhJ0n88kvfbtcO5s6FtLRw4xI5kLIezvoC+DUwJbgQ8C4zqxuDuERSxrhxUKVKYQL5+GO/gJQSiFQEZUoizrm2wP8CL+JPrR0E5JlZtpmdE4P4RJJWQYFf16NbN9/u1MmPh5x1VrhxiZRFmQfWnXOfOOd6AscAffDjE1cA08xsqZn1MbM6+3sOkVQ3YgQcfjhs2+bbixbBW29pwkSpeKI+O8s5t8U59/ci1ckYoBnwBL46ecHMssopTpGk8N13PlHcdJNvX3utP223TZtw4xKJVnmd4rsB2ARsAwx/seDVwAwze93Mjiyn1xGpsB55xJ+uG/HVV34Kd5GKLOokYmaVzay7mU0DlgG3A/nAncBRQAfgHaALMKwcYhWpkNau9dXHH//o2/ff76uPjIxQwxIpF2WegNHMjgN6AT2BusBu/MWBw51z7xXZdTow3cxexV+FLpJyeveGYUU+Qq1fD5qJR5JJWS82fA84F3/I6htgADDCOff1fh42B/hVtAGKVETLl/vrPiKGDoXbbgsvHpFYKWslch5+fqvhwOvOuV0H8Zj/APtLMiJJwzl/yu5rrxX2ff891KwZXkwisVTWJNLSObe8LA9wzi0GFpfxdUQqnFmz4LTTCtvZ2fDb34YXj0g8lCmJlDWBiKSC3bvhzDNh5kzfbtAAcnOhatVQwxKJi4SZCl6kIpoyxU9PEkkgb73ll69VApFUEfYa6yIV0k8/wbHHQl6eb59yCsyYofmuJPWoEhEpo5df9pVGJIF8+inMnq0EIqlJlYjIQdq6FY44wo+BAFxyCUyYoPmuJLWpEhE5CMOG+dN0IwlkyRKYOFEJRESViMh+bNhQ/Arzm26CZ54JLx6RRKNKRKQU/fsXTyCrVyuBiJSkJCJSwpo1/jDVww/7dv/+/kr0Jk3CjUskEYWaRMws3cxmmtkCM1tiZg+VuP9vZra1SLuqmY01sxwzm2FmGfGOWZLbTTdB06aF7Q0b4E9/Ci0ckYQXdiWyHejgnDsRyAQ6mdkZAMGCViVXSLwe2OScOw4YAjwWz2AleS1d6quPESN8e9gwX30UXf9DRPYWahJxXqTSqBzcnJml4ddvv6fEQ7oCo4PtV4GOZjo/RqLnHHTuDK1b+3ZaGvzwA9xyS7hxiVQUYVcimFmamc0H1gNTnHMzgN7AROfcNyV2bwSsAXDO7QS24Nc0KfmcvcxstpnNzs/Pj+03IBXWZ5/BYYfBm2/69tixsHMn1KgRblwiFUnop/gG08lnmlltYLyZnQP8Gr9uSbTPOQIYAZCVleXKI05JHrt2wamnwrx5vt20KaxYAVWqhBuXSEUUeiUS4ZzbjF+r5DzgOCDHzHKB6maWE+y2FmgCYGaVgCOAjfGPViqqt9+GSpUKE8iUKbBqlRKISLRCrUTMrB6wwzm32cyqARcAjznnGhTZZ2swkA4wEbgG+BToBkx1zqnSkAPavh2aNYN163z7jDPg44/94SwRiV7Yf0INgWlmthCYhR8TeWM/+48E6gaVyZ3AvXGIUSq4f/0L0tMLE8isWX7SRCUQkUMXaiXinFsInHSAfWoU2d6GHy8ROaAffoBatQrbl10Gr76q+a5EypM+i0lSGjq0eAL5/HMYN04JRKS8hX52lkh5Wr8e6tcvbPfuDX//e3jxiCQ7VSKSNPr1K55A8vKUQERiTUlEKrzcXH+Y6s9/9u1HHvFXojdqFGpYIilBh7OkQrvuOvjHPwrbGzfCkUeGF49IqlElsi/Z2ZCR4c8BzcjwbUkoixb56iOSQJ591lcfSiAi8aVKpKTsbOjVCwoKfHvVKt8G6NEjvLgE8ImiUyeYPNm3q1Xz07VXrx5uXCKpSpVISf36FSaQiIIC3y+hilxhHkkg48b5H40SiEh4VImUtHp12fol5nbtgsxMWLzYt487zq//UblyuHGJiCqRvRVd1u5g+iWm3njDT5gYSSBTp/oZd5VARBKDkkhJjz669/GR6tV9v8TNtm1+kPySS3z7nHN8RXLeeeHGJSLFKYmU1KOHXyO1WTN/+k+zZr6tQfW4GT3aD5hv2uTbc+bA++9rwkSRRKQxkX3p0UNJIwRbtkDt2oXt7t39DLya70okcemznSSEv/61eAJZsQL+/W8lEJFEp0pEQrVuHTRoUNi+4w544onw4hGRslElIqHp27d4Avn6ayUQkYpGSUTi7ssv/WGqwYN9e+BAfyV6w4bhxiUiZafDWRJXV15ZfCqyTZuKj4WISMWiSkTiYsECX31EEsjIkb76UAIRqdhCTSJmlm5mM81sgZktMbOHgv5sM1tuZovNbJSZVQ76zcz+ZmY5ZrbQzE4OM345MOf8BYKZmb5dq5af7+q668KNS0TKR9iVyHagg3PuRCAT6GRmZwDZwAlAW6AacEOw/y+AFsGtF/B03COWg/bBB/4CwenTffv11/21INWqhRqWiJSjUMdEnHMO2Bo0Kwc355ybFNnHzGYCjYNmV2BM8LjPzKy2mTV0zn0Tz7hl/3buhNat4YsvfPuEE/z6H5U0AieSdMKuRDCzNDObD6wHpjjnZhS5rzJwFfB20NUIWFPk4XlBX8nn7GVms81sdn5+fuyCl728/rqfHDGSQN5/H5YtUwIRSVahJxHn3C7nXCa+2jjNzNoUuXs48IFz7sMyPucI51yWcy6rXr165RmulOLHH6FGDfjVr3y7QwfYvdtPnCgiySv0JBLhnNsMTAM6AZhZf6AecGeR3dYCTYq0Gwd9EqKRI/1Ex//9r2/Pnw/vvacpS0RSQdhnZ9Uzs9rBdjXgAuBzM7sBuAi4wjm3u8hDJgJXB2dpnQFs0XhIeDZv9onihuC0hyuv9GdjnXhiuHGJSPyEfaS6ITDazNLwCe1l59wbZrYTWAV8av7j7GvOuYeBScDFQA5QAFwbTtgycCDcd19he+VK+NnPwotHRMIR9tlZC4GT9tG/z7iCs7JujXVcUrqvv4ZGRU5luOceeOyx8OIRkXCFXYlIBXLHHfDkk4Xtb7+F+vXDi0dEwpcwA+uSuFas8GMfkQQyeLAf+1ACERFVIlIq5/zqgi+/XNi3ZYufukREBFSJSCm8CQyyAAANOElEQVTmzvVTlkQSyJgxPqkogYhIUapEpJjIBYIff+zbRx0Fa9ZAenq4cYlIYlIlIntMmwZpaYUJ5I03ID9fCURESqdKRNixA/7nf+Crr3y7XTt/OCstLdy4RCTxqRJJcePGQZUqhQnk44/9AlJKICJyMFSJpKj//hfq1oXt2327UyeYNEnzXYlI2agSSUHPPutn3I0kkEWL4K23lEBEpOxUiaSQ777z1UfEddf5GXhFRKKlSiRFDBhQPIF89ZUSiIgcOlUiSW7tWmjcuLB9//3w6KPhxSMiyUVJJIn17g3DhhW2168HLfQoIuVJh7OS0PLlfpA8kkCGDvVTliiBiEh5UyWSRJyDyy6D118v7Pv+e6hZM7yYRCS5qRJJErNm+QkTIwkkO9snFSUQEYklVSIV3O7dcOaZMHOmbzds6M+8qlo13LhEJDWoEqnApkzx05NEEsjbb/vla5VARCReQk0iZpZuZjPNbIGZLTGzh4L+5mY2w8xyzGysmVUJ+qsG7Zzg/oww4w/LTz9BkyZw4YW+fcopsHMnXHRRuHGJSOoJuxLZDnRwzp0IZAKdzOwM4DFgiHPuOGATcH2w//XApqB/SLBfShk71lcaeXm+/dlnMHu2JkwUkXCEmkSctzVoVg5uDugAvBr0jwYuDba7Bm2C+zuapcaMT1u3+oHz7t19u0sXPx5y+unhxiUiqS3sSgQzSzOz+cB6YAqwEtjsnNsZ7JIHNAq2GwFrAIL7twB1KcHMepnZbDObnZ+fH+tvIeaeesqfZeWcby9dChMmaMJEEQlf6EnEObfLOZcJNAZOA04oh+cc4ZzLcs5l1avAV9ht2OATxe9/79s33eQTScuW4cYlIhIRehKJcM5tBqYBZwK1zSxy+nFjYG2wvRZoAhDcfwSwMc6hxsWDDxa/wnz1anjmmfDiERHZl7DPzqpnZrWD7WrABcAyfDLpFux2DTAh2J4YtAnun+pc5CBPcli92lcfAwb4dv/+vvpo0iTcuERE9iXsiw0bAqPNLA2f0F52zr1hZkuBl8zsEWAeEJm0fCTwopnlAN8B3cMIOlZ69YLnnitsb9hQfPp2EZFEE2oScc4tBE7aR/+X+PGRkv3bgF/HIbS4WroUWrcubA8bBrfcEl48IiIHK+xKJKU5B5dcAm++6duVKsHmzXD44eHGJSJysBJmYD3VfPqpv+4jkkDGjoUdO5RARKRiUSUSZ7t2wamnwrx5vt20KaxYAVWqhBuXiEg0VInE0Vtv+UNWkQQyZQqsWqUEIiIVlyqRONi+3Vcc69f79plnwkcf+cNZIiIVmf6NxVh2NqSnFyaQWbPgk0+UQEQkOagSiZHvv4cjjihsX345vPKK5rsSkeSiz8Mx8OSTxRPI8uXw6qtKICKSfFSJlKP166F+/cJ2797w97+HF4+ISKypEikn999fPIHk5SmBiEjyUxI5RLm5/jDVX/7i24884q9Eb9Rovw8TEUkKOpx1CHr2hNGjC9vffQd16oQWjohI3KkSicKiRb76iCSQESN89aEEIiKpRpVIGTgHF13krzQHqFbNT9devXq4cYmIhEWVyEH6+GN/gWAkgYwbBwUFSiAiktpUiZTQoAGsW1f6/cceC8uWQeXK8YtJRCRRqRIpYX8JZNo0yMlRAhERiVASKYNzzw07AhGRxKIkIiIiUQs1iZhZEzObZmZLzWyJmfUJ+jPN7DMzm29ms83stKDfzOxvZpZjZgvN7OQw4xcRSXVhD6zvBO5yzs01s5rAHDObAjwOPOSce8vMLg7a5wK/AFoEt9OBp4OvIiISglArEefcN865ucH2D8AyoBHggFrBbkcAXwfbXYExzvsMqG1mDcszpqLzXx1Mv4hIKgu7EtnDzDKAk4AZwO3AO2Y2GJ/ozgp2awSsKfKwvKDvmxLP1QvoBdC0adMyxfHtt2UOXUQkZSXEwLqZ1QDGAbc7574HbgbucM41Ae4ARpbl+ZxzI5xzWc65rHr16pV/wCIiAiRAEjGzyvgEku2cey3ovgaIbL8CnBZsrwWaFHl446BPRERCEPbZWYavMpY5554octfXwM+D7Q7AimB7InB1cJbWGcAW51yxQ1kiIhI/YY+JtAeuAhaZ2fyg737gRmComVUCthGMbwCTgIuBHKAAuDa+4YqISFGhJhHn3EdAaSuPn7KP/R1wa0yDEhGRg2b+/3LyMrN8YFWUDz8K2FCO4ZQXxVU2iqvsEjU2xVU2hxJXM+fcAc9MSvokcijMbLZzLivsOEpSXGWjuMouUWNTXGUTj7hCPztLREQqLiURERGJmpLI/o0IO4BSKK6yUVxll6ixKa6yiXlcGhMREZGoqRIREZGopVwSMbNRZrbezBaX6P+9mX0erGvyeJH++4L1S5ab2UVF+jsFfTlmdm8s4jKzscGaKvPNLLfIBZlhx1Xm9V7M7BozWxHcrolRXCea2admtsjM/mNmtYrcF6/3q7Q1co40synB9z/FzOoE/XF5z/YT16+D9m4zyyrxmJi/Z/uJa1Dw97jQzMabWe0EiWtAENN8M5tsZscE/aH+HIvcf5eZOTM7Km5xOedS6gacA5wMLC7Sdx7wLlA1aB8dfG0FLACqAs2BlUBacFsJ/AyoEuzTqrzjKnH/X4EHEyEuYDLwi2D7YmB6ke238BeQngHMCPqPBL4MvtYJtuvEIK5ZwM+D7euAASG8Xw2Bk4PtmsAXwes/Dtwb9N8LPBbP92w/cbUE/geYDmQV2T8u79l+4roQqBT0P1bk/Qo7rlpF9rkNeCYRfo5BuwnwDv66uKPiFVfKVSLOuQ+A70p03wwMdM5tD/ZZH/R3BV5yzm13zn2Fn27ltOCW45z70jn3E/BSsG95xwXsmWPsN8C/EySusq73chEwxTn3nXNuEzAF6BSDuI4HPgi2pwCXF4krXu9XaWvkdAVGB7uNBi4tElvM37PS4nLOLXPOLd/HQ+Lynu0nrsnOuZ3Bbp/hJ1tNhLi+L7Lb4fi/hUhcof0cg7uHAPcUiSkucaVcEinF8cDZZjbDzN43s1OD/tLWLymtP1bOBtY55yITUYYd1+3AIDNbAwwG7kuQuJZQ+I/j1xTO+BxKXFZ8jZz6rnCy0G+ByDJncY+tRFylSaS4rsN/mk6IuMzs0eB3vwfwYCLEZWZdgbXOuQUldot5XEoiXiV8WXcG0Bd4Ofj0nyiuoLAKSQSHtN5LDF0H3GJmc/Cl/k9hBWJ7r5Gzh/PHE0I5LXJ/cYWptLjMrB9+Ge3sRInLOdcv+N3PBnqHHRf+/bmfwoQWV0oiXh7wWlDyzQR24+ecKW39krita2J+JuPLgLFFusOOq6zrvcQlLufc5865C51zp+CT7sow4rJ9r5GzLjiMQPA1csg0brGVEldpQo/LzHoCnYEeQeJNiLiKyKbwkGmYcR2LHx9aYGa5wWvMNbMGcYkrmoGUin4DMig+IPs74OFg+3h8mWdAa4oP4n2JH8CrFGw3p3AQr3V5xxX0dQLeL9EXalz447DnBtsdgTnB9i8pPog30xUO4n2FH8CrE2wfGYO4IidEHAaMAa6L9/sVfO9jgCdL9A+i+MD64/F8z0qLq8j90yk+sB6X92w/71cnYClQL4zf/f3E1aLI9u+BVxPp5xjsk0vhwHrM4zqkP+KKeMN/Qv0G2IGvQK4Pfun+CSwG5gIdiuzfD/+JdjnBGUlB/8X4MyNWAv1iEVfQ/wLwu33sH1pcwP8Cc4I/1BnAKUV+wYcFr72I4v+UrsMPguYA18Yorj7B9/4FMJDgYto4v1//iz9UtRCYH9wuBuoC7+EXWHs38gcbr/dsP3H9Knj/tgPrgHfi+Z7tJ64c/Ie5SN8zCRLXOPz/iYXAf/CD7aH/HEvsk0thEol5XLpiXUREoqYxERERiZqSiIiIRE1JREREoqYkIiIiUVMSERGRqCmJiIhI1JREROLAzK4Ppuh+az/7vBnsc0s8YxM5FEoiInHgnBsJTAQ6mdmtJe83s5sJpu12zg2Pd3wi0dLFhiJxYmZH4692Phy/JsTyoP94YB7wI9DGOfdteFGKlI0qEZE4cX6dmhuB6sA/zaxSMMHmP4O+XkogUtFUCjsAkVTinJtgZqPw8xZFpu4+FXjBHXhmXZGEo8NZInFmZjXxk1c2DbrWAO2cX6lOpELR4SyROAuSxcMUrg1+sxKIVFRKIiJxZmbVgD8U6fp1WLGIHColEZH4exw4ARiKXw/iOjO7JNyQRKKjMRGRODKzC4G38af6ngq0AGYDm/Gn924IMTyRMlMlIhInZnYk8A/8aoxXOue2O+cWA38E6gNPhxmfSDSURETi5xngGOAB59zCIv1/BT4EupnZlaFEJhIlHc4SiQMzuwoYA3wAnOec213i/ub4dbN3Am2dc3nxj1Kk7JRERGLMzJriE4ThrwdZVcp+NwDPAVOAi5z+OKUCUBIREZGoaUxERESipiQiIiJRUxIREZGoKYmIiEjUlERERCRqSiIiIhI1JREREYmakoiIiERNSURERKKmJCIiIlH7f1gLS3bZv0mkAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "predict = model(Variable(x_train))\n",
    "predict = predict.data.numpy()\n",
    "plt.plot(x_train.numpy(), y_train.numpy(), 'ro', label='Original data')\n",
    "plt.plot(x_train.numpy(), predict, 'b-s', label='Fitting Line')\n",
    "plt.xlabel('X', fontsize= 20)\n",
    "plt.ylabel('y', fontsize= 20)\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "使用pytorch建立卷积神经网络并处理MNIST数据。\n",
    "https://computational-communication.com/pytorch-mnist/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<!--NAVIGATION-->\n",
    "< [In-Depth: Decision Trees and Random Forests](09.08-Random-Forests.ipynb) | [Contents](Index.ipynb) |[In-Depth: Neural Network Advanced](09.10.neural_network_advanced.ipynb)>"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "170px"
   },
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
